2024-02-10T06:27:43.008 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-10T06:27:43.008 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} --archive /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555158 --name yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555158/orig.config.yaml 2024-02-10T06:27:43.047 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T06:27:43.124 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555158 branch: main description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: yweinste@redhat.com first_in_suite: false job_id: '7555158' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 1610 sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 targets: smithi008.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF2aUKsBzj/VJ8KwhUWNGSToLomQpKaI08g8aqfn3rmIfB0jYyt1/LwmV9XVr8zpa3p+ZtDByLoVxoE517vK6Y0= smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNrtBko9bMq22M2Y9UwpUu0ycburNrK0a2irK58FScViUNaLzP/mmR34uUX9Y6Hc/8xdrXUcoIreMYz8geIsGjA= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_19:44:16 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-02-10T06:27:43.124 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa; will attempt to use it 2024-02-10T06:27:43.125 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks 2024-02-10T06:27:43.125 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-10T06:27:43.127 INFO:teuthology.task.internal:Checking packages... 2024-02-10T06:27:43.151 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90' 2024-02-10T06:27:43.151 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T06:27:43.152 INFO:teuthology.packaging:ref: None 2024-02-10T06:27:43.152 INFO:teuthology.packaging:tag: None 2024-02-10T06:27:43.152 INFO:teuthology.packaging:branch: main 2024-02-10T06:27:43.152 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:27:43.152 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-02-10T06:27:43.404 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1361.gf8e4e331 2024-02-10T06:27:43.406 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-10T06:27:43.412 INFO:teuthology.task.internal:no buildpackages task found 2024-02-10T06:27:43.412 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-10T06:27:43.445 INFO:teuthology.task.internal:Saving configuration 2024-02-10T06:27:43.459 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-10T06:27:43.470 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T06:27:43.501 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi008.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555158', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 06:22:27.726461', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF2aUKsBzj/VJ8KwhUWNGSToLomQpKaI08g8aqfn3rmIfB0jYyt1/LwmV9XVr8zpa3p+ZtDByLoVxoE517vK6Y0='} 2024-02-10T06:27:43.525 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555158', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 06:22:27.728218', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNrtBko9bMq22M2Y9UwpUu0ycburNrK0a2irK58FScViUNaLzP/mmR34uUX9Y6Hc/8xdrXUcoIreMYz8geIsGjA='} 2024-02-10T06:27:43.526 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-10T06:27:43.532 INFO:teuthology.task.internal:roles: ubuntu@smithi008.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-02-10T06:27:43.532 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-02-10T06:27:43.532 INFO:teuthology.run_tasks:Running task console_log... 2024-02-10T06:27:43.618 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1f38fa81f0>, signals=[15]) 2024-02-10T06:27:43.618 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-10T06:27:43.624 INFO:teuthology.task.internal:Opening connections... 2024-02-10T06:27:43.624 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T06:27:43.626 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:27:43.730 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-02-10T06:27:43.731 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:27:43.840 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-10T06:27:43.846 DEBUG:teuthology.orchestra.run.smithi008:> uname -m 2024-02-10T06:27:43.864 INFO:teuthology.orchestra.run.smithi008.stdout:x86_64 2024-02-10T06:27:43.864 DEBUG:teuthology.orchestra.run.smithi008:> cat /etc/os-release 2024-02-10T06:27:43.920 INFO:teuthology.orchestra.run.smithi008.stdout:NAME="CentOS Stream" 2024-02-10T06:27:43.920 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION="9" 2024-02-10T06:27:43.921 INFO:teuthology.orchestra.run.smithi008.stdout:ID="centos" 2024-02-10T06:27:43.921 INFO:teuthology.orchestra.run.smithi008.stdout:ID_LIKE="rhel fedora" 2024-02-10T06:27:43.921 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_ID="9" 2024-02-10T06:27:43.921 INFO:teuthology.orchestra.run.smithi008.stdout:PLATFORM_ID="platform:el9" 2024-02-10T06:27:43.921 INFO:teuthology.orchestra.run.smithi008.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T06:27:43.921 INFO:teuthology.orchestra.run.smithi008.stdout:ANSI_COLOR="0;31" 2024-02-10T06:27:43.921 INFO:teuthology.orchestra.run.smithi008.stdout:LOGO="fedora-logo-icon" 2024-02-10T06:27:43.921 INFO:teuthology.orchestra.run.smithi008.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T06:27:43.921 INFO:teuthology.orchestra.run.smithi008.stdout:HOME_URL="https://centos.org/" 2024-02-10T06:27:43.921 INFO:teuthology.orchestra.run.smithi008.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T06:27:43.921 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T06:27:43.921 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T06:27:43.922 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2024-02-10T06:27:43.943 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-02-10T06:27:43.960 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-02-10T06:27:43.960 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-02-10T06:27:44.017 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2024-02-10T06:27:44.017 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2024-02-10T06:27:44.018 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2024-02-10T06:27:44.018 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2024-02-10T06:27:44.018 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2024-02-10T06:27:44.018 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2024-02-10T06:27:44.018 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T06:27:44.018 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-02-10T06:27:44.018 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2024-02-10T06:27:44.018 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T06:27:44.018 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2024-02-10T06:27:44.018 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T06:27:44.018 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T06:27:44.018 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T06:27:44.019 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-02-10T06:27:44.042 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-10T06:27:44.050 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-10T06:27:44.070 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-10T06:27:44.070 DEBUG:teuthology.orchestra.run.smithi008:> test '!' -e /home/ubuntu/cephtest 2024-02-10T06:27:44.073 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-02-10T06:27:44.089 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-10T06:27:44.095 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-10T06:27:44.095 DEBUG:teuthology.orchestra.run.smithi008:> test -z $(ls -A /var/lib/ceph) 2024-02-10T06:27:44.130 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-02-10T06:27:44.163 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-10T06:27:44.220 INFO:teuthology.run_tasks:Running task kernel... 2024-02-10T06:27:44.234 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-10T06:27:44.234 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-10T06:27:44.234 DEBUG:teuthology.orchestra.run.smithi008:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T06:27:44.235 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T06:27:44.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:27:44.251 DEBUG:teuthology.orchestra.run.smithi008:> uname -r 2024-02-10T06:27:44.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:27:44.253 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-02-10T06:27:44.310 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-361.el9.x86_64 2024-02-10T06:27:44.310 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.14.0-361.el9.x86_64 2024-02-10T06:27:44.310 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2024-02-10T06:27:44.311 INFO:teuthology.orchestra.run.smithi008.stdout:5.14.0-361.el9.x86_64 2024-02-10T06:27:44.311 INFO:teuthology.task.kernel:Running kernel on smithi008: 5.14.0-361.el9.x86_64 2024-02-10T06:27:44.312 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum install -y kernel 2024-02-10T06:27:46.907 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS-9 - AppStream 6.5 MB/s | 6.7 MB 00:01 2024-02-10T06:27:47.710 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 8.7 MB/s | 6.7 MB 00:00 2024-02-10T06:27:48.382 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-02-10T06:27:49.496 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 3.3 MB/s | 2.0 MB 00:00 2024-02-10T06:27:49.898 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - BaseOS 6.8 MB/s | 8.0 MB 00:01 2024-02-10T06:27:50.690 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 9.4 MB/s | 8.0 MB 00:00 2024-02-10T06:27:52.957 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 19 MB 00:01 2024-02-10T06:27:54.690 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 6.5 MB/s | 19 MB 00:02 2024-02-10T06:27:57.250 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 5.9 MB 00:00 2024-02-10T06:27:58.834 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-02-10T06:28:03.855 INFO:teuthology.orchestra.run.smithi008.stdout:Extra Packages for Enterprise Linux 4.2 MB/s | 20 MB 00:04 2024-02-10T06:28:08.877 INFO:teuthology.orchestra.run.smithi008.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-02-10T06:28:11.250 INFO:teuthology.orchestra.run.smithi008.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T06:28:11.251 INFO:teuthology.orchestra.run.smithi008.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T06:28:11.340 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T06:28:11.356 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T06:28:11.356 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T06:28:11.356 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T06:28:11.356 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T06:28:11.356 INFO:teuthology.orchestra.run.smithi008.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T06:28:11.356 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T06:28:11.356 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T06:28:11.356 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T06:28:11.357 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T06:28:11.357 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:28:11.357 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T06:28:11.357 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T06:28:11.357 INFO:teuthology.orchestra.run.smithi008.stdout:Install 4 Packages 2024-02-10T06:28:11.357 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:28:11.357 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 99 M 2024-02-10T06:28:11.357 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 122 M 2024-02-10T06:28:11.358 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T06:28:11.703 INFO:teuthology.orchestra.run.smithi008.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 17 MB/s | 6.0 MB 00:00 2024-02-10T06:28:12.179 INFO:teuthology.orchestra.run.smithi008.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 25 MB/s | 21 MB 00:00 2024-02-10T06:28:12.804 INFO:teuthology.orchestra.run.smithi008.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 30 MB/s | 33 MB 00:01 2024-02-10T06:28:13.246 INFO:teuthology.orchestra.run.smithi008.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 21 MB/s | 39 MB 00:01 2024-02-10T06:28:13.248 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:28:13.249 INFO:teuthology.orchestra.run.smithi008.stdout:Total 52 MB/s | 99 MB 00:01 2024-02-10T06:28:13.249 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T06:28:13.711 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T06:28:13.712 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T06:28:14.813 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T06:28:14.814 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T06:28:14.858 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 369 kB/s | 5.9 MB 00:16 2024-02-10T06:28:16.452 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 15 kB 00:00 2024-02-10T06:28:17.355 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T06:28:17.715 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 18 MB/s | 20 MB 00:01 2024-02-10T06:28:18.025 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T06:28:18.155 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T06:28:21.083 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T06:28:21.261 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T06:28:22.722 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-02-10T06:28:25.107 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T06:28:25.107 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T06:28:25.233 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-02-10T06:28:25.248 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-02-10T06:28:25.248 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-02-10T06:28:25.248 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-02-10T06:28:25.248 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-02-10T06:28:25.248 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T06:28:25.248 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-02-10T06:28:25.248 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T06:28:25.249 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T06:28:25.249 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T06:28:25.249 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:28:25.249 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-02-10T06:28:25.249 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-02-10T06:28:25.249 INFO:teuthology.orchestra.run.smithi029.stdout:Install 4 Packages 2024-02-10T06:28:25.249 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:28:25.250 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 99 M 2024-02-10T06:28:25.250 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 122 M 2024-02-10T06:28:25.250 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-02-10T06:28:25.826 INFO:teuthology.orchestra.run.smithi029.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 10 MB/s | 6.0 MB 00:00 2024-02-10T06:28:26.329 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T06:28:26.377 INFO:teuthology.orchestra.run.smithi029.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 18 MB/s | 21 MB 00:01 2024-02-10T06:28:26.432 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:28:27.272 INFO:teuthology.orchestra.run.smithi029.stdout:(3/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 19 MB/s | 39 MB 00:02 2024-02-10T06:28:27.978 INFO:teuthology.orchestra.run.smithi029.stdout:(4/4): kernel-modules-core-5.14.0-419.el9.x86_6 16 MB/s | 33 MB 00:02 2024-02-10T06:28:27.981 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:28:27.981 INFO:teuthology.orchestra.run.smithi029.stdout:Total 36 MB/s | 99 MB 00:02 2024-02-10T06:28:27.981 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-02-10T06:28:28.456 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-02-10T06:28:28.456 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-02-10T06:28:29.497 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-02-10T06:28:29.498 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-02-10T06:28:31.520 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:28:32.059 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-02-10T06:28:32.853 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T06:28:33.042 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T06:28:35.944 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T06:28:36.243 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T06:28:41.308 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T06:28:41.446 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:28:46.528 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:29:02.232 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:29:02.232 INFO:teuthology.orchestra.run.smithi008.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T06:29:02.232 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:29:02.304 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:29:02.725 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:29:02.725 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T06:29:02.725 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T06:29:02.725 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T06:29:03.184 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:29:03.184 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:29:03.184 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T06:29:03.184 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T06:29:03.184 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T06:29:03.184 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T06:29:03.184 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T06:29:03.184 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:29:03.184 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T06:29:03.679 DEBUG:teuthology.orchestra.run.smithi008:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T06:29:03.821 INFO:teuthology.orchestra.run.smithi008.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T06:29:03.822 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T06:29:03.822 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T06:29:03.822 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T06:29:03.822 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T06:29:03.822 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-10T06:29:03.822 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi008.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T06:29:03.822 DEBUG:teuthology.orchestra.run.smithi008:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T06:29:03.935 INFO:teuthology.orchestra.run.smithi008.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T06:29:03.935 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T06:29:03.935 DEBUG:teuthology.orchestra.run.smithi008:> sudo rpm -qi grub2-tools 2024-02-10T06:29:03.980 INFO:teuthology.orchestra.run.smithi008.stdout:Name : grub2-tools 2024-02-10T06:29:03.980 INFO:teuthology.orchestra.run.smithi008.stdout:Epoch : 1 2024-02-10T06:29:03.980 INFO:teuthology.orchestra.run.smithi008.stdout:Version : 2.06 2024-02-10T06:29:03.980 INFO:teuthology.orchestra.run.smithi008.stdout:Release : 68.el9 2024-02-10T06:29:03.980 INFO:teuthology.orchestra.run.smithi008.stdout:Architecture: x86_64 2024-02-10T06:29:03.981 INFO:teuthology.orchestra.run.smithi008.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T06:29:03.981 INFO:teuthology.orchestra.run.smithi008.stdout:Group : Unspecified 2024-02-10T06:29:03.981 INFO:teuthology.orchestra.run.smithi008.stdout:Size : 8264254 2024-02-10T06:29:03.981 INFO:teuthology.orchestra.run.smithi008.stdout:License : GPLv3+ 2024-02-10T06:29:03.981 INFO:teuthology.orchestra.run.smithi008.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T06:29:03.981 INFO:teuthology.orchestra.run.smithi008.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T06:29:03.981 INFO:teuthology.orchestra.run.smithi008.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T06:29:03.981 INFO:teuthology.orchestra.run.smithi008.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T06:29:03.981 INFO:teuthology.orchestra.run.smithi008.stdout:Packager : builder@centos.org 2024-02-10T06:29:03.981 INFO:teuthology.orchestra.run.smithi008.stdout:Vendor : CentOS 2024-02-10T06:29:03.981 INFO:teuthology.orchestra.run.smithi008.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T06:29:03.981 INFO:teuthology.orchestra.run.smithi008.stdout:Summary : Support tools for GRUB. 2024-02-10T06:29:03.981 INFO:teuthology.orchestra.run.smithi008.stdout:Description : 2024-02-10T06:29:03.981 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:29:03.982 INFO:teuthology.orchestra.run.smithi008.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T06:29:03.982 INFO:teuthology.orchestra.run.smithi008.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T06:29:03.982 INFO:teuthology.orchestra.run.smithi008.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T06:29:03.982 INFO:teuthology.orchestra.run.smithi008.stdout:hardware devices. 2024-02-10T06:29:03.982 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:29:03.982 INFO:teuthology.orchestra.run.smithi008.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T06:29:03.983 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T06:29:03.983 INFO:teuthology.task.kernel:Updating grub on smithi008 to boot 5.14.0-419.el9.x86_64 2024-02-10T06:29:03.983 DEBUG:teuthology.orchestra.run.smithi008:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T06:29:04.546 INFO:teuthology.orchestra.run.smithi008.stderr:Generating grub configuration file ... 2024-02-10T06:29:05.894 INFO:teuthology.orchestra.run.smithi008.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T06:29:05.931 INFO:teuthology.orchestra.run.smithi008.stderr:done 2024-02-10T06:29:05.933 DEBUG:teuthology.orchestra.run.smithi008:> mktemp 2024-02-10T06:29:05.948 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.RAKNMYrzuP 2024-02-10T06:29:05.949 DEBUG:teuthology.orchestra.run.smithi008:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RAKNMYrzuP 2024-02-10T06:29:06.037 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.RAKNMYrzuP 2024-02-10T06:29:06.140 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.RAKNMYrzuP is 6KB 2024-02-10T06:29:06.189 DEBUG:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.RAKNMYrzuP 2024-02-10T06:29:06.204 DEBUG:teuthology.orchestra.run.smithi008:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T06:29:06.272 INFO:teuthology.orchestra.run.smithi008.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T06:29:06.272 INFO:teuthology.orchestra.run.smithi008.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T06:29:06.272 INFO:teuthology.orchestra.run.smithi008.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T06:29:06.272 INFO:teuthology.orchestra.run.smithi008.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T06:29:06.273 DEBUG:teuthology.orchestra.run.smithi008:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T06:29:06.421 DEBUG:teuthology.orchestra.run.smithi008:> sudo shutdown -r now 2024-02-10T06:29:19.495 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:29:19.495 INFO:teuthology.orchestra.run.smithi029.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T06:29:19.496 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:29:19.566 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:29:20.051 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:29:20.051 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T06:29:20.051 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T06:29:20.052 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T06:29:20.528 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T06:29:20.528 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:29:20.528 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-02-10T06:29:20.528 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T06:29:20.528 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T06:29:20.529 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T06:29:20.529 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T06:29:20.529 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:29:20.529 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-02-10T06:29:21.264 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T06:29:21.462 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T06:29:21.462 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T06:29:21.462 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T06:29:21.462 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T06:29:21.463 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T06:29:21.463 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-10T06:29:21.463 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T06:29:21.463 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T06:29:21.575 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T06:29:21.576 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T06:29:21.576 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2024-02-10T06:29:21.616 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2024-02-10T06:29:21.616 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2024-02-10T06:29:21.616 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.06 2024-02-10T06:29:21.616 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 68.el9 2024-02-10T06:29:21.616 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2024-02-10T06:29:21.617 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T06:29:21.617 INFO:teuthology.orchestra.run.smithi029.stdout:Group : Unspecified 2024-02-10T06:29:21.617 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 8264254 2024-02-10T06:29:21.617 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2024-02-10T06:29:21.617 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T06:29:21.617 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T06:29:21.617 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T06:29:21.617 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T06:29:21.617 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : builder@centos.org 2024-02-10T06:29:21.617 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2024-02-10T06:29:21.617 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T06:29:21.617 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2024-02-10T06:29:21.617 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2024-02-10T06:29:21.617 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:29:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T06:29:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T06:29:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T06:29:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2024-02-10T06:29:21.618 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:29:21.618 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T06:29:21.618 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T06:29:21.618 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 5.14.0-419.el9.x86_64 2024-02-10T06:29:21.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T06:29:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2024-02-10T06:29:23.949 INFO:teuthology.orchestra.run.smithi029.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T06:29:23.975 INFO:teuthology.orchestra.run.smithi029.stderr:done 2024-02-10T06:29:23.977 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-02-10T06:29:24.003 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.css9Q4i2Be 2024-02-10T06:29:24.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.css9Q4i2Be 2024-02-10T06:29:24.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.css9Q4i2Be 2024-02-10T06:29:24.222 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.css9Q4i2Be is 6KB 2024-02-10T06:29:24.271 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.css9Q4i2Be 2024-02-10T06:29:24.286 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T06:29:24.356 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T06:29:24.357 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T06:29:24.357 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T06:29:24.357 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T06:29:24.358 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T06:29:24.521 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2024-02-10T06:29:36.437 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-10T06:29:36.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T06:29:36.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:29:54.540 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-10T06:29:54.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-02-10T06:29:54.541 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:30:11.005 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-02-10T06:30:13.056 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-02-10T06:30:20.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T06:30:20.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:30:22.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-02-10T06:30:22.060 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:30:23.068 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-02-10T06:30:25.116 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-02-10T06:30:35.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T06:30:35.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:30:37.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-02-10T06:30:37.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:30:38.140 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-02-10T06:30:53.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T06:30:53.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:31:09.166 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-02-10T06:31:10.204 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-02-10T06:31:10.204 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-419.el9.x86_64"... 2024-02-10T06:31:10.204 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-02-10T06:31:10.218 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-419.el9.x86_64 2024-02-10T06:31:10.219 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T06:31:10.219 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T06:31:10.220 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T06:31:11.220 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-10T06:31:11.220 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T06:31:11.345 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2024-02-10T06:31:11.393 DEBUG:teuthology.parallel:result is None 2024-02-10T06:31:11.676 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-02-10T06:31:29.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T06:31:29.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:31:30.050 DEBUG:teuthology.orchestra.run.smithi008:> true 2024-02-10T06:31:30.477 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T06:31:30.477 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-419.el9.x86_64"... 2024-02-10T06:31:30.477 DEBUG:teuthology.orchestra.run.smithi008:> uname -r 2024-02-10T06:31:30.491 INFO:teuthology.orchestra.run.smithi008.stdout:5.14.0-419.el9.x86_64 2024-02-10T06:31:30.491 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T06:31:30.491 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T06:31:30.491 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T06:31:31.493 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-10T06:31:31.493 DEBUG:teuthology.orchestra.run.smithi008:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T06:31:31.576 INFO:teuthology.orchestra.run.smithi008.stdout:ttyS1 2024-02-10T06:31:31.613 DEBUG:teuthology.parallel:result is None 2024-02-10T06:31:31.613 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-10T06:31:31.621 INFO:teuthology.task.internal:Creating test directory... 2024-02-10T06:31:31.621 DEBUG:teuthology.orchestra.run.smithi008:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T06:31:31.623 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T06:31:31.641 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-10T06:31:31.670 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-10T06:31:31.763 INFO:teuthology.task.internal:Creating archive directory... 2024-02-10T06:31:31.763 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T06:31:31.766 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T06:31:31.801 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-10T06:31:31.846 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-10T06:31:31.847 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T06:31:31.849 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T06:31:31.889 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T06:31:31.893 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T06:31:31.903 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T06:31:31.906 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T06:31:31.907 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-10T06:31:31.929 INFO:teuthology.task.internal:Configuring sudo... 2024-02-10T06:31:31.929 DEBUG:teuthology.orchestra.run.smithi008:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T06:31:31.931 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T06:31:31.974 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-10T06:31:32.006 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-10T06:31:32.006 DEBUG:teuthology.orchestra.run.smithi008:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T06:31:32.008 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T06:31:32.030 DEBUG:teuthology.orchestra.run.smithi008:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T06:31:32.105 DEBUG:teuthology.orchestra.run.smithi008:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T06:31:32.146 DEBUG:teuthology.orchestra.run.smithi008:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T06:31:32.227 DEBUG:teuthology.orchestra.run.smithi008:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T06:31:32.269 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T06:31:32.269 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T06:31:32.335 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T06:31:32.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T06:31:32.434 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T06:31:32.519 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T06:31:32.562 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T06:31:32.562 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T06:31:32.631 DEBUG:teuthology.orchestra.run.smithi008:> sudo service rsyslog restart 2024-02-10T06:31:32.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-02-10T06:31:32.693 INFO:teuthology.orchestra.run.smithi008.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T06:31:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T06:31:33.101 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-10T06:31:33.109 INFO:teuthology.task.internal:Starting timer... 2024-02-10T06:31:33.110 INFO:teuthology.run_tasks:Running task pcp... 2024-02-10T06:31:33.141 INFO:teuthology.run_tasks:Running task selinux... 2024-02-10T06:31:33.198 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-10T06:31:33.198 DEBUG:teuthology.orchestra.run.smithi008:> sudo service auditd rotate 2024-02-10T06:31:33.263 INFO:teuthology.orchestra.run.smithi008.stdout:Rotating logs: 2024-02-10T06:31:33.265 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-02-10T06:31:33.330 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-02-10T06:31:33.332 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-10T06:31:33.332 DEBUG:teuthology.orchestra.run.smithi008:> /usr/sbin/getenforce 2024-02-10T06:31:33.357 INFO:teuthology.orchestra.run.smithi008.stdout:Permissive 2024-02-10T06:31:33.357 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-02-10T06:31:33.386 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-02-10T06:31:33.386 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi008.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi029.front.sepia.ceph.com': 'permissive'} 2024-02-10T06:31:33.386 DEBUG:teuthology.orchestra.run.smithi008:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T06:31:33.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:31:33.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T06:31:33.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:31:33.457 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-10T06:31:33.457 DEBUG:teuthology.orchestra.run.smithi008:> sudo /usr/sbin/setenforce permissive 2024-02-10T06:31:33.509 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-02-10T06:31:33.543 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-10T06:31:33.554 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-10T06:31:33.572 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-10T06:31:33.573 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi008.front.sepia.ceph.com,smithi029.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-10T06:35:56.966 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi008.front.sepia.ceph.com'), Remote(name='ubuntu@smithi029.front.sepia.ceph.com')] 2024-02-10T06:35:56.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T06:35:56.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:35:57.060 DEBUG:teuthology.orchestra.run.smithi008:> true 2024-02-10T06:35:57.138 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T06:35:57.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-02-10T06:35:57.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:35:57.227 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-02-10T06:35:57.316 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-02-10T06:35:57.316 INFO:teuthology.run_tasks:Running task clock... 2024-02-10T06:35:57.327 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-10T06:35:57.327 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T06:35:57.328 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T06:35:57.330 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T06:35:57.330 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T06:35:57.358 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T06:35:57.375 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T06:35:57.388 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T06:35:57.404 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: ntpd: command not found 2024-02-10T06:35:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T06:35:57.418 INFO:teuthology.orchestra.run.smithi008.stdout:506 Cannot talk to daemon 2024-02-10T06:35:57.435 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-02-10T06:35:57.435 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T06:35:57.448 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-02-10T06:35:57.451 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T06:35:57.465 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T06:35:57.481 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T06:35:57.503 INFO:teuthology.orchestra.run.smithi008.stderr:bash: line 1: ntpq: command not found 2024-02-10T06:35:57.506 INFO:teuthology.orchestra.run.smithi008.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T06:35:57.506 INFO:teuthology.orchestra.run.smithi008.stdout:=============================================================================== 2024-02-10T06:35:57.506 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:35:57.506 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:35:57.506 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:35:57.506 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:35:57.528 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-02-10T06:35:57.531 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T06:35:57.532 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-02-10T06:35:57.532 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:35:57.532 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:35:57.532 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:35:57.532 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:35:57.533 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-10T06:35:57.630 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-10T06:35:57.630 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-02-10T06:35:57.630 INFO:tasks.cephadm:Cluster fsid is a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:35:57.630 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-10T06:35:57.631 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-10T06:35:57.631 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi008': '172.21.15.8', 'mon.smithi029': '172.21.15.29'} 2024-02-10T06:35:57.631 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-10T06:35:57.631 DEBUG:teuthology.orchestra.run.smithi008:> sudo hostname $(hostname -s) 2024-02-10T06:35:57.658 DEBUG:teuthology.orchestra.run.smithi029:> sudo hostname $(hostname -s) 2024-02-10T06:35:57.685 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-02-10T06:35:57.686 DEBUG:teuthology.orchestra.run.smithi008:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T06:35:57.922 INFO:teuthology.orchestra.run.smithi008.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Feb 10 06:35 /home/ubuntu/cephtest/cephadm 2024-02-10T06:35:57.922 DEBUG:teuthology.orchestra.run.smithi029:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T06:35:58.086 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Feb 10 06:35 /home/ubuntu/cephtest/cephadm 2024-02-10T06:35:58.086 DEBUG:teuthology.orchestra.run.smithi008:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T06:35:58.101 DEBUG:teuthology.orchestra.run.smithi029:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T06:35:58.125 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-02-10T06:35:58.125 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-02-10T06:35:58.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-02-10T06:35:58.984 INFO:teuthology.orchestra.run.smithi008.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-10T06:35:59.078 INFO:teuthology.orchestra.run.smithi029.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-10T06:36:17.344 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-02-10T06:36:17.344 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-02-10T06:36:17.344 INFO:teuthology.orchestra.run.smithi029.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-02-10T06:36:17.345 INFO:teuthology.orchestra.run.smithi029.stdout: "repo_digests": [ 2024-02-10T06:36:17.345 INFO:teuthology.orchestra.run.smithi029.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-02-10T06:36:17.345 INFO:teuthology.orchestra.run.smithi029.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-02-10T06:36:17.345 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-02-10T06:36:17.345 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-02-10T06:36:19.985 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:36:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-02-10T06:36:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-02-10T06:36:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: "repo_digests": [ 2024-02-10T06:36:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-02-10T06:36:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-02-10T06:36:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T06:36:19.985 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:36:20.004 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /etc/ceph 2024-02-10T06:36:20.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph 2024-02-10T06:36:20.076 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 777 /etc/ceph 2024-02-10T06:36:20.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /etc/ceph 2024-02-10T06:36:20.146 INFO:tasks.cephadm:Writing seed config... 2024-02-10T06:36:20.147 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-10T06:36:20.148 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-10T06:36:20.148 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-10T06:36:20.148 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-10T06:36:20.148 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-10T06:36:20.148 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-10T06:36:20.148 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-10T06:36:20.148 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-10T06:36:20.149 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T06:36:20.149 DEBUG:teuthology.orchestra.run.smithi008:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-10T06:36:20.163 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a5f941ec-c7de-11ee-95b8-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-02-10T06:36:20.163 DEBUG:teuthology.orchestra.run.smithi008:mon.smithi008> sudo journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mon.smithi008.service 2024-02-10T06:36:20.205 INFO:tasks.cephadm:Bootstrapping... 2024-02-10T06:36:20.205 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.8 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:36:20.404 INFO:teuthology.orchestra.run.smithi008.stderr:-------------------------------------------------------------------------------- 2024-02-10T06:36:20.404 INFO:teuthology.orchestra.run.smithi008.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'a5f941ec-c7de-11ee-95b8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.8', '--skip-admin-label'] 2024-02-10T06:36:20.428 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: 4.9.3-dev 2024-02-10T06:36:20.429 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying podman|docker is present... 2024-02-10T06:36:20.450 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: 4.9.3-dev 2024-02-10T06:36:20.452 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying lvm2 is present... 2024-02-10T06:36:20.452 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying time synchronization is in place... 2024-02-10T06:36:20.459 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T06:36:20.465 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-02-10T06:36:20.472 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: enabled 2024-02-10T06:36:20.477 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: active 2024-02-10T06:36:20.478 INFO:teuthology.orchestra.run.smithi008.stderr:Unit chronyd.service is enabled and running 2024-02-10T06:36:20.478 INFO:teuthology.orchestra.run.smithi008.stderr:Repeating the final host check... 2024-02-10T06:36:20.499 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: 4.9.3-dev 2024-02-10T06:36:20.501 INFO:teuthology.orchestra.run.smithi008.stderr:podman (/bin/podman) version 4.9.3 is present 2024-02-10T06:36:20.501 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl is present 2024-02-10T06:36:20.501 INFO:teuthology.orchestra.run.smithi008.stderr:lvcreate is present 2024-02-10T06:36:20.507 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T06:36:20.513 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-02-10T06:36:20.519 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: enabled 2024-02-10T06:36:20.524 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: active 2024-02-10T06:36:20.525 INFO:teuthology.orchestra.run.smithi008.stderr:Unit chronyd.service is enabled and running 2024-02-10T06:36:20.525 INFO:teuthology.orchestra.run.smithi008.stderr:Host looks OK 2024-02-10T06:36:20.525 INFO:teuthology.orchestra.run.smithi008.stderr:Cluster fsid: a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:36:20.525 INFO:teuthology.orchestra.run.smithi008.stderr:Acquiring lock 139730296053136 on /run/cephadm/a5f941ec-c7de-11ee-95b8-87774f69a715.lock 2024-02-10T06:36:20.526 INFO:teuthology.orchestra.run.smithi008.stderr:Lock 139730296053136 acquired on /run/cephadm/a5f941ec-c7de-11ee-95b8-87774f69a715.lock 2024-02-10T06:36:20.526 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying IP 172.21.15.8 port 3300 ... 2024-02-10T06:36:20.526 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying IP 172.21.15.8 port 6789 ... 2024-02-10T06:36:20.526 INFO:teuthology.orchestra.run.smithi008.stderr:Base mon IP is 172.21.15.8, final addrv is [v2:172.21.15.8:3300,v1:172.21.15.8:6789] 2024-02-10T06:36:20.528 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.8 metric 100 2024-02-10T06:36:20.529 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.8 metric 100 2024-02-10T06:36:20.531 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-10T06:36:20.531 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-02-10T06:36:20.531 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-02-10T06:36:20.534 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-10T06:36:20.534 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-10T06:36:20.534 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-10T06:36:20.534 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-10T06:36:20.534 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: inet6 fe80::ae1f:6bff:fef8:25fe/64 scope link noprefixroute 2024-02-10T06:36:20.534 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-10T06:36:20.535 INFO:teuthology.orchestra.run.smithi008.stderr:Mon IP `172.21.15.8` is in CIDR network `172.21.0.0/20` 2024-02-10T06:36:20.535 INFO:teuthology.orchestra.run.smithi008.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-10T06:36:20.535 INFO:teuthology.orchestra.run.smithi008.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-10T06:36:20.639 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-02-10T06:36:21.133 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Getting image source signatures 2024-02-10T06:36:21.133 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-02-10T06:36:21.133 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-02-10T06:36:21.133 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-02-10T06:36:21.133 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-02-10T06:36:21.134 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-02-10T06:36:21.134 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-02-10T06:36:21.364 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Writing manifest to image destination 2024-02-10T06:36:21.491 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-02-10T06:36:21.899 INFO:teuthology.orchestra.run.smithi008.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-02-10T06:36:22.483 INFO:teuthology.orchestra.run.smithi008.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-02-10T06:36:22.483 INFO:teuthology.orchestra.run.smithi008.stderr:Extracting ceph user uid/gid from container image... 2024-02-10T06:36:22.907 INFO:teuthology.orchestra.run.smithi008.stderr:stat: 167 167 2024-02-10T06:36:23.388 INFO:teuthology.orchestra.run.smithi008.stderr:Creating initial keys... 2024-02-10T06:36:23.745 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph-authtool: AQBnGcdl6H1+JxAAYpns8z+SVlZH8XJKvzI+WA== 2024-02-10T06:36:24.642 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph-authtool: AQBoGcdlAgtAIRAAP4o3kz08U39qeV9ZzexrjQ== 2024-02-10T06:36:25.681 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph-authtool: AQBpGcdlglcJJRAA0bAlNszpP8Xv6xmutYgIKQ== 2024-02-10T06:36:26.189 INFO:teuthology.orchestra.run.smithi008.stderr:Creating initial monmap... 2024-02-10T06:36:26.571 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T06:36:26.571 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-02-10T06:36:26.571 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:36:26.571 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T06:36:26.977 INFO:teuthology.orchestra.run.smithi008.stderr:monmaptool for smithi008 [v2:172.21.15.8:3300,v1:172.21.15.8:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T06:36:26.977 INFO:teuthology.orchestra.run.smithi008.stderr:setting min_mon_release = octopus 2024-02-10T06:36:26.977 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: set fsid to a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:36:26.977 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T06:36:26.977 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-02-10T06:36:26.977 INFO:teuthology.orchestra.run.smithi008.stderr:Creating mon... 2024-02-10T06:36:27.957 INFO:teuthology.orchestra.run.smithi008.stderr:create mon.smithi008 on 2024-02-10T06:36:28.171 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-10T06:36:28.317 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a5f941ec-c7de-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-a5f941ec-c7de-11ee-95b8-87774f69a715.target. 2024-02-10T06:36:28.317 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-a5f941ec-c7de-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-a5f941ec-c7de-11ee-95b8-87774f69a715.target. 2024-02-10T06:36:28.608 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Failed to reset failed state of unit ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mon.smithi008.service: Unit ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mon.smithi008.service not loaded. 2024-02-10T06:36:28.615 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Created symlink /etc/systemd/system/ceph-a5f941ec-c7de-11ee-95b8-87774f69a715.target.wants/ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mon.smithi008.service → /etc/systemd/system/ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@.service. 2024-02-10T06:36:29.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:29 smithi008 ceph-mon[24305]: mon.smithi008 is new leader, mons smithi008 in quorum (ranks 0) 2024-02-10T06:36:29.261 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: disabled 2024-02-10T06:36:29.268 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-02-10T06:36:29.269 INFO:teuthology.orchestra.run.smithi008.stderr:firewalld.service is not enabled 2024-02-10T06:36:29.269 INFO:teuthology.orchestra.run.smithi008.stderr:Not possible to enable service . firewalld.service is not available 2024-02-10T06:36:29.269 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mon to start... 2024-02-10T06:36:29.269 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mon... 2024-02-10T06:36:29.514 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:29 smithi008 bash[24287]: 8743ab52cc74f5979e50c0648b5a4b573f4fec4273284236f2f1b64e5d7664ab 2024-02-10T06:36:29.514 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:29 smithi008 systemd[1]: Started Ceph mon.smithi008 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:36:29.807 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: cluster: 2024-02-10T06:36:29.807 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: id: a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:36:29.807 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-10T06:36:29.807 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-02-10T06:36:29.807 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: services: 2024-02-10T06:36:29.807 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi008 (age 0.577237s) 2024-02-10T06:36:29.808 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-10T06:36:29.808 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-10T06:36:29.808 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-02-10T06:36:29.808 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: data: 2024-02-10T06:36:29.808 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-10T06:36:29.808 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-10T06:36:29.808 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-10T06:36:29.808 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: pgs: 2024-02-10T06:36:29.808 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-02-10T06:36:30.365 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:30 smithi008 ceph-mon[24305]: mon.smithi008 is new leader, mons smithi008 in quorum (ranks 0) 2024-02-10T06:36:30.365 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:30 smithi008 ceph-mon[24305]: monmap e1: 1 mons at {smithi008=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0]} 2024-02-10T06:36:30.366 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:30 smithi008 ceph-mon[24305]: fsmap 2024-02-10T06:36:30.366 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:30 smithi008 ceph-mon[24305]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T06:36:30.366 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:30 smithi008 ceph-mon[24305]: mgrmap e1: no daemons active 2024-02-10T06:36:30.366 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:30 smithi008 ceph-mon[24305]: from='client.? 172.21.15.8:0/275295521' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T06:36:30.368 INFO:teuthology.orchestra.run.smithi008.stderr:mon is available 2024-02-10T06:36:30.368 INFO:teuthology.orchestra.run.smithi008.stderr:Assimilating anything we can from ceph.conf... 2024-02-10T06:36:30.845 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-02-10T06:36:30.845 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [global] 2024-02-10T06:36:30.845 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: fsid = a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:36:30.845 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.8:3300,v1:172.21.15.8:6789] 2024-02-10T06:36:30.845 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-10T06:36:30.845 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-10T06:36:30.845 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-10T06:36:30.845 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-10T06:36:30.845 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-02-10T06:36:30.845 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [mgr] 2024-02-10T06:36:30.846 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-10T06:36:30.846 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-02-10T06:36:30.846 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [osd] 2024-02-10T06:36:30.846 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-10T06:36:30.846 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-10T06:36:31.404 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:31 smithi008 ceph-mon[24305]: from='client.? 172.21.15.8:0/389600625' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T06:36:31.405 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:31 smithi008 ceph-mon[24305]: from='client.? 172.21.15.8:0/389600625' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T06:36:31.407 INFO:teuthology.orchestra.run.smithi008.stderr:Generating new minimal ceph.conf... 2024-02-10T06:36:32.397 INFO:teuthology.orchestra.run.smithi008.stderr:Restarting the monitor... 2024-02-10T06:36:32.439 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:32 smithi008 ceph-mon[24305]: from='client.? 172.21.15.8:0/2090273116' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:36:32.439 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:32 smithi008 systemd[1]: Stopping Ceph mon.smithi008 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:36:32.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:32 smithi008 bash[24524]: Error: no container with name or ID "ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon.smithi008" found: no such container 2024-02-10T06:36:32.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:32 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008[24301]: 2024-02-10T06:36:32.563+0000 7fba4e1f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi008 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:36:32.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:32 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008[24301]: 2024-02-10T06:36:32.563+0000 7fba4e1f5700 -1 mon.smithi008@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T06:36:32.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:32 smithi008 podman[24537]: 2024-02-10 06:36:32.624805483 +0000 UTC m=+0.176399417 container died 8743ab52cc74f5979e50c0648b5a4b573f4fec4273284236f2f1b64e5d7664ab (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, GIT_CLEAN=True, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, vcs-type=git, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., name=centos-stream, release=754) 2024-02-10T06:36:33.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 podman[24537]: 2024-02-10 06:36:33.021272085 +0000 UTC m=+0.572866027 container cleanup 8743ab52cc74f5979e50c0648b5a4b573f4fec4273284236f2f1b64e5d7664ab (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, ceph=True, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., vcs-type=git, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, version=8, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-02-10T06:36:33.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 bash[24537]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008 2024-02-10T06:36:33.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 bash[24565]: Error: no container with name or ID "ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon.smithi008" found: no such container 2024-02-10T06:36:33.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 podman[24549]: 2024-02-10 06:36:33.247217006 +0000 UTC m=+0.626393306 container remove 8743ab52cc74f5979e50c0648b5a4b573f4fec4273284236f2f1b64e5d7664ab (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , release=754, name=centos-stream, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, vcs-type=git, io.buildah.version=1.19.8, vendor=Red Hat, Inc., distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, architecture=x86_64, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, ceph=True, GIT_CLEAN=True) 2024-02-10T06:36:33.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mon.smithi008.service: Deactivated successfully. 2024-02-10T06:36:33.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 systemd[1]: Stopped Ceph mon.smithi008 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:36:33.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 systemd[1]: Starting Ceph mon.smithi008 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:36:33.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 podman[24627]: 2024-02-10 06:36:33.481372537 +0000 UTC m=+0.113150132 container create 061312907e4b95e3ed2a34007b7f6a69c9cb7067f0a83d4ba6fbd579d146331f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., name=centos-stream, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream) 2024-02-10T06:36:33.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 podman[24627]: 2024-02-10 06:36:33.386525243 +0000 UTC m=+0.018302846 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-02-10T06:36:33.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 podman[24627]: 2024-02-10 06:36:33.691769501 +0000 UTC m=+0.323547116 container init 061312907e4b95e3ed2a34007b7f6a69c9cb7067f0a83d4ba6fbd579d146331f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, vendor=Red Hat, Inc., RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, GIT_CLEAN=True, name=centos-stream, vcs-type=git, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, release=754) 2024-02-10T06:36:33.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 podman[24627]: 2024-02-10 06:36:33.69455042 +0000 UTC m=+0.326328019 container start 061312907e4b95e3ed2a34007b7f6a69c9cb7067f0a83d4ba6fbd579d146331f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, build-date=2022-05-03T08:36:31.336870, name=centos-stream, io.openshift.expose-services=, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, architecture=x86_64, release=754, com.redhat.component=centos-stream-container, vcs-type=git) 2024-02-10T06:36:33.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T06:36:33.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-02-10T06:36:33.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: pidfile_write: ignore empty --pid-file 2024-02-10T06:36:33.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: load: jerasure load: lrc 2024-02-10T06:36:33.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: RocksDB version: 6.15.5 2024-02-10T06:36:33.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T06:36:33.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Compile date Apr 18 2022 2024-02-10T06:36:33.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: DB SUMMARY 2024-02-10T06:36:33.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: DB Session ID: ESJVH62WV66SHT6RAX2X 2024-02-10T06:36:33.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: CURRENT file: CURRENT 2024-02-10T06:36:33.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T06:36:33.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-02-10T06:36:33.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi008/store.db dir, Total Num: 1, files: 000008.sst 2024-02-10T06:36:33.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi008/store.db: 000010.log size: 82673 ; 2024-02-10T06:36:33.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.error_if_exists: 0 2024-02-10T06:36:33.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.create_if_missing: 0 2024-02-10T06:36:33.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.paranoid_checks: 1 2024-02-10T06:36:33.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T06:36:33.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.env: 0x5614fb0c6860 2024-02-10T06:36:33.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.fs: Posix File System 2024-02-10T06:36:33.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.info_log: 0x5614fcad9fa0 2024-02-10T06:36:33.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T06:36:33.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.statistics: (nil) 2024-02-10T06:36:33.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.use_fsync: 0 2024-02-10T06:36:33.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_log_file_size: 0 2024-02-10T06:36:33.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T06:36:33.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T06:36:33.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T06:36:33.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T06:36:33.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.allow_fallocate: 1 2024-02-10T06:36:33.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T06:36:33.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T06:36:33.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.use_direct_reads: 0 2024-02-10T06:36:33.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T06:36:33.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T06:36:33.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.db_log_dir: 2024-02-10T06:36:33.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi008/store.db 2024-02-10T06:36:33.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T06:36:33.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T06:36:33.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T06:36:33.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T06:36:33.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T06:36:33.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T06:36:33.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T06:36:33.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T06:36:33.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.write_buffer_manager: 0x5614fcbca330 2024-02-10T06:36:33.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T06:36:33.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T06:36:33.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T06:36:33.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T06:36:33.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T06:36:33.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T06:36:33.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T06:36:33.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T06:36:33.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T06:36:33.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.unordered_write: 0 2024-02-10T06:36:33.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T06:36:33.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T06:36:33.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T06:36:33.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T06:36:33.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.row_cache: None 2024-02-10T06:36:33.781 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.wal_filter: None 2024-02-10T06:36:33.781 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T06:36:33.781 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T06:36:33.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.preserve_deletes: 0 2024-02-10T06:36:33.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.two_write_queues: 0 2024-02-10T06:36:33.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T06:36:33.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.atomic_flush: 0 2024-02-10T06:36:33.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T06:36:33.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T06:36:33.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T06:36:33.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.log_readahead_size: 0 2024-02-10T06:36:33.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T06:36:33.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T06:36:33.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T06:36:33.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T06:36:33.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T06:36:33.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T06:36:33.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_background_jobs: 2 2024-02-10T06:36:33.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_background_compactions: -1 2024-02-10T06:36:33.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_subcompactions: 1 2024-02-10T06:36:33.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T06:36:33.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T06:36:33.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T06:36:33.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T06:36:33.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T06:36:33.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T06:36:33.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T06:36:33.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T06:36:33.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_open_files: -1 2024-02-10T06:36:33.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T06:36:33.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T06:36:33.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T06:36:33.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T06:36:33.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_background_flushes: -1 2024-02-10T06:36:33.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Compression algorithms supported: 2024-02-10T06:36:33.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T06:36:33.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: kZSTD supported: 0 2024-02-10T06:36:33.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: kXpressCompression supported: 0 2024-02-10T06:36:33.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T06:36:33.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: kLZ4Compression supported: 1 2024-02-10T06:36:33.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: kBZip2Compression supported: 0 2024-02-10T06:36:33.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: kZlibCompression supported: 1 2024-02-10T06:36:33.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: kSnappyCompression supported: 1 2024-02-10T06:36:33.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T06:36:33.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi008/store.db/MANIFEST-000009 2024-02-10T06:36:33.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-02-10T06:36:33.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T06:36:33.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.merge_operator: 2024-02-10T06:36:33.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compaction_filter: None 2024-02-10T06:36:33.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T06:36:33.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T06:36:33.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T06:36:33.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T06:36:33.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5614fcaa5d00) 2024-02-10T06:36:33.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: cache_index_and_filter_blocks: 1 2024-02-10T06:36:33.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T06:36:33.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T06:36:33.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: pin_top_level_index_and_filter: 1 2024-02-10T06:36:33.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_type: 0 2024-02-10T06:36:33.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: data_block_index_type: 0 2024-02-10T06:36:33.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_shortening: 1 2024-02-10T06:36:33.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T06:36:33.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: hash_index_allow_collision: 1 2024-02-10T06:36:33.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: checksum: 1 2024-02-10T06:36:33.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: no_block_cache: 0 2024-02-10T06:36:33.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache: 0x5614fcb10170 2024-02-10T06:36:33.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_name: BinnedLRUCache 2024-02-10T06:36:33.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_options: 2024-02-10T06:36:33.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: capacity : 536870912 2024-02-10T06:36:33.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: num_shard_bits : 4 2024-02-10T06:36:33.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: strict_capacity_limit : 0 2024-02-10T06:36:33.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: high_pri_pool_ratio: 0.000 2024-02-10T06:36:33.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_compressed: (nil) 2024-02-10T06:36:33.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: persistent_cache: (nil) 2024-02-10T06:36:33.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_size: 4096 2024-02-10T06:36:33.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_size_deviation: 10 2024-02-10T06:36:33.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_restart_interval: 16 2024-02-10T06:36:33.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_block_restart_interval: 1 2024-02-10T06:36:33.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: metadata_block_size: 4096 2024-02-10T06:36:33.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: partition_filters: 0 2024-02-10T06:36:33.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: use_delta_encoding: 1 2024-02-10T06:36:33.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T06:36:33.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: whole_key_filtering: 1 2024-02-10T06:36:33.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: verify_compression: 0 2024-02-10T06:36:33.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: read_amp_bytes_per_bit: 0 2024-02-10T06:36:33.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: format_version: 4 2024-02-10T06:36:33.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: enable_index_compression: 1 2024-02-10T06:36:33.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_align: 0 2024-02-10T06:36:33.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T06:36:33.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T06:36:33.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compression: NoCompression 2024-02-10T06:36:33.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T06:36:33.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T06:36:33.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T06:36:33.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.num_levels: 7 2024-02-10T06:36:33.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T06:36:33.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T06:36:33.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T06:36:33.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T06:36:33.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T06:36:33.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T06:36:33.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T06:36:33.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:36:33.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T06:36:33.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T06:36:33.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T06:36:33.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T06:36:33.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T06:36:33.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T06:36:33.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:36:33.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T06:36:33.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T06:36:33.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T06:36:33.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T06:36:33.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T06:36:33.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T06:36:33.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T06:36:33.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T06:36:33.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T06:36:33.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T06:36:33.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T06:36:33.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T06:36:33.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T06:36:33.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T06:36:33.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T06:36:33.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T06:36:33.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T06:36:33.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T06:36:33.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.arena_block_size: 4194304 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.table_properties_collectors: 2024-02-10T06:36:33.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.inplace_update_support: 0 2024-02-10T06:36:33.792 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T06:36:33.792 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T06:36:33.792 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T06:36:33.792 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T06:36:33.792 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.bloom_locality: 0 2024-02-10T06:36:33.792 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.max_successive_merges: 0 2024-02-10T06:36:33.792 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T06:36:33.792 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T06:36:33.792 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T06:36:33.792 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T06:36:33.792 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.ttl: 2592000 2024-02-10T06:36:33.792 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T06:36:33.792 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.enable_blob_files: false 2024-02-10T06:36:33.792 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.min_blob_size: 0 2024-02-10T06:36:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T06:36:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T06:36:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T06:36:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T06:36:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi008/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T06:36:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-02-10T06:36:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-02-10T06:36:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707546993719734, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-02-10T06:36:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-02-10T06:36:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T06:36:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707546993720502, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79355, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77619, "index_size": 224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9884, "raw_average_key_size": 47, "raw_value_size": 71928, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 209, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707546993, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "0e262ca0-0a67-455c-94b9-bc71eda80113", "db_session_id": "ESJVH62WV66SHT6RAX2X"}} 2024-02-10T06:36:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-02-10T06:36:33.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707546993720834, "job": 1, "event": "recovery_finished"} 2024-02-10T06:36:33.794 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi008/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T06:36:33.794 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5614fcaf6a80 2024-02-10T06:36:33.794 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: DB pointer 0x5614fcb06000 2024-02-10T06:36:33.794 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-02-10T06:36:33.794 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-10T06:36:33.794 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** DB Stats ** 2024-02-10T06:36:33.794 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:36:33.794 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T06:36:33.794 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T06:36:33.794 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:36:33.794 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T06:36:33.795 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T06:36:33.795 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:36:33.795 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-02-10T06:36:33.795 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-02-10T06:36:33.795 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T06:36:33.795 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:36:33.795 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: L0 2/0 79.29 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.6 0.00 0.00 1 0.001 0 0 2024-02-10T06:36:33.795 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Sum 2/0 79.29 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.6 0.00 0.00 1 0.001 0 0 2024-02-10T06:36:33.795 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.6 0.00 0.00 1 0.001 0 0 2024-02-10T06:36:33.795 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-02-10T06:36:33.795 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-02-10T06:36:33.795 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T06:36:33.795 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:36:33.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 138.6 0.00 0.00 1 0.001 0 0 2024-02-10T06:36:33.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:36:33.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T06:36:33.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T06:36:33.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T06:36:33.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T06:36:33.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T06:36:33.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative compaction: 0.00 GB write, 25.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:36:33.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval compaction: 0.00 GB write, 25.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:36:33.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T06:36:33.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-02-10T06:36:33.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T06:36:33.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-02-10T06:36:33.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-02-10T06:36:33.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T06:36:33.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:36:33.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: L0 2/0 79.29 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.6 0.00 0.00 1 0.001 0 0 2024-02-10T06:36:33.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Sum 2/0 79.29 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.6 0.00 0.00 1 0.001 0 0 2024-02-10T06:36:33.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T06:36:33.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-02-10T06:36:33.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-02-10T06:36:33.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T06:36:33.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:36:33.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 138.6 0.00 0.00 1 0.001 0 0 2024-02-10T06:36:33.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:36:33.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T06:36:33.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T06:36:33.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T06:36:33.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative compaction: 0.00 GB write, 24.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: starting mon.smithi008 rank 0 at public addrs [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] at bind addrs [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi008 fsid a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: mon.smithi008@-1(???) e1 preinit fsid a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: mon.smithi008@-1(???).mds e1 new map 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: mon.smithi008@-1(???).mds e1 print_map 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: e1 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: legacy client fscid: -1 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: No filesystems configured 2024-02-10T06:36:33.798 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: mon.smithi008@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T06:36:33.799 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: mon.smithi008@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T06:36:33.799 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: mon.smithi008@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T06:36:33.799 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: mon.smithi008@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T06:36:33.799 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: mon.smithi008@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-10T06:36:33.799 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: expand_channel_meta expand map: {default=false} 2024-02-10T06:36:33.799 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: expand_channel_meta from 'false' to 'false' 2024-02-10T06:36:33.799 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: expand_channel_meta expanded map: {default=false} 2024-02-10T06:36:33.799 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: expand_channel_meta expand map: {default=info} 2024-02-10T06:36:33.799 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: expand_channel_meta from 'info' to 'info' 2024-02-10T06:36:33.799 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: expand_channel_meta expanded map: {default=info} 2024-02-10T06:36:33.799 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: expand_channel_meta expand map: {default=daemon} 2024-02-10T06:36:33.800 INFO:teuthology.orchestra.run.smithi008.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-10T06:36:33.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: expand_channel_meta from 'daemon' to 'daemon' 2024-02-10T06:36:33.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: expand_channel_meta expanded map: {default=daemon} 2024-02-10T06:36:33.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: expand_channel_meta expand map: {default=debug} 2024-02-10T06:36:33.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: expand_channel_meta from 'debug' to 'debug' 2024-02-10T06:36:33.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: expand_channel_meta expanded map: {default=debug} 2024-02-10T06:36:33.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 bash[24627]: 061312907e4b95e3ed2a34007b7f6a69c9cb7067f0a83d4ba6fbd579d146331f 2024-02-10T06:36:33.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 systemd[1]: Started Ceph mon.smithi008 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:36:34.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: mon.smithi008 is new leader, mons smithi008 in quorum (ranks 0) 2024-02-10T06:36:34.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: monmap e1: 1 mons at {smithi008=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0]} 2024-02-10T06:36:34.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: fsmap 2024-02-10T06:36:34.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T06:36:34.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:33 smithi008 ceph-mon[24646]: mgrmap e1: no daemons active 2024-02-10T06:36:34.776 INFO:teuthology.orchestra.run.smithi008.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-10T06:36:34.777 INFO:teuthology.orchestra.run.smithi008.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:36:34.777 INFO:teuthology.orchestra.run.smithi008.stderr:Creating mgr... 2024-02-10T06:36:34.777 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying port 9283 ... 2024-02-10T06:36:34.943 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Failed to reset failed state of unit ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mgr.smithi008.pxifbt.service: Unit ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mgr.smithi008.pxifbt.service not loaded. 2024-02-10T06:36:34.950 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Created symlink /etc/systemd/system/ceph-a5f941ec-c7de-11ee-95b8-87774f69a715.target.wants/ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mgr.smithi008.pxifbt.service → /etc/systemd/system/ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@.service. 2024-02-10T06:36:35.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:35 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1802176260' entity='client.admin' 2024-02-10T06:36:35.573 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: disabled 2024-02-10T06:36:35.580 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-02-10T06:36:35.581 INFO:teuthology.orchestra.run.smithi008.stderr:firewalld.service is not enabled 2024-02-10T06:36:35.581 INFO:teuthology.orchestra.run.smithi008.stderr:Not possible to enable service . firewalld.service is not available 2024-02-10T06:36:35.588 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: disabled 2024-02-10T06:36:35.595 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-02-10T06:36:35.596 INFO:teuthology.orchestra.run.smithi008.stderr:firewalld.service is not enabled 2024-02-10T06:36:35.596 INFO:teuthology.orchestra.run.smithi008.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-10T06:36:35.596 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mgr to start... 2024-02-10T06:36:35.596 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mgr... 2024-02-10T06:36:36.085 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-02-10T06:36:36.086 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-02-10T06:36:36.086 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsid": "a5f941ec-c7de-11ee-95b8-87774f69a715", 2024-02-10T06:36:36.086 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "health": { 2024-02-10T06:36:36.086 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T06:36:36.086 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T06:36:36.086 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T06:36:36.086 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:36.086 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T06:36:36.086 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T06:36:36.086 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 0 2024-02-10T06:36:36.086 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-02-10T06:36:36.086 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T06:36:36.086 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "smithi008" 2024-02-10T06:36:36.087 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-02-10T06:36:36.087 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-10T06:36:36.087 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T06:36:36.087 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T06:36:36.087 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-10T06:36:36.087 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T06:36:36.087 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:36.087 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T06:36:36.087 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T06:36:36.087 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T06:36:36.087 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T06:36:36.087 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T06:36:36.087 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T06:36:36.090 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T06:36:36.091 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T06:36:36.091 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:36.091 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T06:36:36.091 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T06:36:36.091 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T06:36:36.091 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T06:36:36.091 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T06:36:36.091 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T06:36:36.091 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T06:36:36.091 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T06:36:36.091 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T06:36:36.091 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:36.091 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T06:36:36.091 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T06:36:36.091 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T06:36:36.092 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T06:36:36.092 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:36.092 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T06:36:36.092 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": false, 2024-02-10T06:36:36.092 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T06:36:36.092 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T06:36:36.092 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "iostat", 2024-02-10T06:36:36.092 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "nfs", 2024-02-10T06:36:36.092 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "restful" 2024-02-10T06:36:36.092 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-02-10T06:36:36.092 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-02-10T06:36:36.092 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:36.092 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T06:36:36.092 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T06:36:36.092 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modified": "2024-02-10T06:36:29.228399+0000", 2024-02-10T06:36:36.093 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-02-10T06:36:36.093 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:36.093 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T06:36:36.093 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-02-10T06:36:36.373 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:36 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1006429994' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T06:36:36.636 INFO:teuthology.orchestra.run.smithi008.stderr:mgr not available, waiting (1/15)... 2024-02-10T06:36:41.998 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:41 smithi008 ceph-mon[24646]: Activating manager daemon smithi008.pxifbt 2024-02-10T06:36:41.998 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:41 smithi008 ceph-mon[24646]: mgrmap e2: smithi008.pxifbt(active, starting, since 0.00293614s) 2024-02-10T06:36:41.998 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:41 smithi008 ceph-mon[24646]: from='mgr.14100 172.21.15.8:0/1788139381' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:36:41.998 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:41 smithi008 ceph-mon[24646]: from='mgr.14100 172.21.15.8:0/1788139381' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:36:41.998 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:41 smithi008 ceph-mon[24646]: from='mgr.14100 172.21.15.8:0/1788139381' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:36:41.999 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:41 smithi008 ceph-mon[24646]: from='mgr.14100 172.21.15.8:0/1788139381' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:36:41.999 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:41 smithi008 ceph-mon[24646]: from='mgr.14100 172.21.15.8:0/1788139381' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr metadata", "who": "smithi008.pxifbt", "id": "smithi008.pxifbt"}]: dispatch 2024-02-10T06:36:41.999 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:41 smithi008 ceph-mon[24646]: Manager daemon smithi008.pxifbt is now available 2024-02-10T06:36:41.999 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:41 smithi008 ceph-mon[24646]: from='mgr.14100 172.21.15.8:0/1788139381' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:36:41.999 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:41 smithi008 ceph-mon[24646]: from='mgr.14100 172.21.15.8:0/1788139381' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:36:41.999 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:41 smithi008 ceph-mon[24646]: from='mgr.14100 172.21.15.8:0/1788139381' entity='mgr.smithi008.pxifbt' 2024-02-10T06:36:41.999 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:41 smithi008 ceph-mon[24646]: from='mgr.14100 172.21.15.8:0/1788139381' entity='mgr.smithi008.pxifbt' 2024-02-10T06:36:41.999 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:41 smithi008 ceph-mon[24646]: from='mgr.14100 172.21.15.8:0/1788139381' entity='mgr.smithi008.pxifbt' 2024-02-10T06:36:43.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:42 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1773856476' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T06:36:43.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:42 smithi008 ceph-mon[24646]: mgrmap e3: smithi008.pxifbt(active, since 1.00666s) 2024-02-10T06:36:43.127 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-02-10T06:36:43.127 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-02-10T06:36:43.128 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsid": "a5f941ec-c7de-11ee-95b8-87774f69a715", 2024-02-10T06:36:43.128 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "health": { 2024-02-10T06:36:43.128 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T06:36:43.128 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T06:36:43.128 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T06:36:43.128 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:43.128 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T06:36:43.129 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T06:36:43.129 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 0 2024-02-10T06:36:43.129 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-02-10T06:36:43.129 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T06:36:43.129 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "smithi008" 2024-02-10T06:36:43.129 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-02-10T06:36:43.129 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-02-10T06:36:43.129 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T06:36:43.130 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T06:36:43.130 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-10T06:36:43.130 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T06:36:43.130 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:43.130 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T06:36:43.130 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T06:36:43.130 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T06:36:43.130 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T06:36:43.131 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T06:36:43.131 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T06:36:43.131 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T06:36:43.131 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T06:36:43.131 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:43.131 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T06:36:43.131 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T06:36:43.131 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T06:36:43.132 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T06:36:43.132 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T06:36:43.132 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T06:36:43.132 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T06:36:43.132 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T06:36:43.132 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T06:36:43.132 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:43.132 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T06:36:43.133 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T06:36:43.133 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T06:36:43.133 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T06:36:43.133 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:43.133 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T06:36:43.133 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": false, 2024-02-10T06:36:43.133 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T06:36:43.133 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T06:36:43.133 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "iostat", 2024-02-10T06:36:43.134 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "nfs", 2024-02-10T06:36:43.134 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "restful" 2024-02-10T06:36:43.134 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-02-10T06:36:43.134 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-02-10T06:36:43.134 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:43.134 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T06:36:43.134 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T06:36:43.134 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modified": "2024-02-10T06:36:29.228399+0000", 2024-02-10T06:36:43.134 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-02-10T06:36:43.134 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:43.135 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T06:36:43.135 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-02-10T06:36:45.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:44 smithi008 ceph-mon[24646]: mgrmap e4: smithi008.pxifbt(active, since 2s) 2024-02-10T06:36:48.680 INFO:teuthology.orchestra.run.smithi008.stderr:mgr not available, waiting (2/15)... 2024-02-10T06:36:51.399 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-02-10T06:36:51.399 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-02-10T06:36:51.399 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsid": "a5f941ec-c7de-11ee-95b8-87774f69a715", 2024-02-10T06:36:51.399 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "health": { 2024-02-10T06:36:51.399 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T06:36:51.400 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T06:36:51.400 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T06:36:51.400 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:51.400 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T06:36:51.400 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T06:36:51.400 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 0 2024-02-10T06:36:51.400 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-02-10T06:36:51.400 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T06:36:51.400 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "smithi008" 2024-02-10T06:36:51.400 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-02-10T06:36:51.400 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_age": 17, 2024-02-10T06:36:51.400 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T06:36:51.400 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T06:36:51.400 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-10T06:36:51.401 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T06:36:51.401 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:51.401 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T06:36:51.401 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T06:36:51.401 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T06:36:51.401 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T06:36:51.401 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T06:36:51.404 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T06:36:51.404 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T06:36:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T06:36:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T06:36:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T06:36:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T06:36:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T06:36:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T06:36:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T06:36:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T06:36:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T06:36:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T06:36:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T06:36:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T06:36:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T06:36:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T06:36:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T06:36:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": true, 2024-02-10T06:36:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T06:36:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T06:36:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "iostat", 2024-02-10T06:36:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "nfs", 2024-02-10T06:36:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "restful" 2024-02-10T06:36:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-02-10T06:36:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-02-10T06:36:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T06:36:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T06:36:51.407 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modified": "2024-02-10T06:36:29.228399+0000", 2024-02-10T06:36:51.407 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-02-10T06:36:51.407 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-02-10T06:36:51.407 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T06:36:51.407 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-02-10T06:36:51.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:51 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/2534096567' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T06:36:51.986 INFO:teuthology.orchestra.run.smithi008.stderr:mgr is available 2024-02-10T06:36:52.606 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-02-10T06:36:52.606 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [global] 2024-02-10T06:36:52.606 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: fsid = a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:36:52.606 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-10T06:36:52.606 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-10T06:36:52.606 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-10T06:36:52.607 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-10T06:36:52.607 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-02-10T06:36:52.607 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [mgr] 2024-02-10T06:36:52.607 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-10T06:36:52.607 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-02-10T06:36:52.607 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [osd] 2024-02-10T06:36:52.607 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-10T06:36:52.607 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-10T06:36:52.941 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:52 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/2481542874' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T06:36:53.168 INFO:teuthology.orchestra.run.smithi008.stderr:Enabling cephadm module... 2024-02-10T06:36:55.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:54 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/3356861609' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-10T06:36:56.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:55 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/3356861609' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-10T06:36:56.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:55 smithi008 ceph-mon[24646]: mgrmap e5: smithi008.pxifbt(active, since 13s) 2024-02-10T06:36:57.132 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-02-10T06:36:57.132 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-10T06:36:57.132 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": true, 2024-02-10T06:36:57.132 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "active_name": "smithi008.pxifbt", 2024-02-10T06:36:57.132 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-10T06:36:57.132 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-02-10T06:36:57.429 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:36:57 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/723636010' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T06:36:57.740 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for the mgr to restart... 2024-02-10T06:36:57.740 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mgr epoch 5... 2024-02-10T06:37:01.805 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-02-10T06:37:01.805 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-10T06:37:01.805 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "initialized": true 2024-02-10T06:37:01.805 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-02-10T06:37:02.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: Active manager daemon smithi008.pxifbt restarted 2024-02-10T06:37:02.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: Activating manager daemon smithi008.pxifbt 2024-02-10T06:37:02.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: osdmap e2: 0 total, 0 up, 0 in 2024-02-10T06:37:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: mgrmap e6: smithi008.pxifbt(active, starting, since 0.0527572s) 2024-02-10T06:37:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:37:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr metadata", "who": "smithi008.pxifbt", "id": "smithi008.pxifbt"}]: dispatch 2024-02-10T06:37:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:37:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:37:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:37:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: Manager daemon smithi008.pxifbt is now available 2024-02-10T06:37:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:37:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:37:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:37:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:37:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:01 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:37:02.371 INFO:teuthology.orchestra.run.smithi008.stderr:mgr epoch 5 is available 2024-02-10T06:37:02.372 INFO:teuthology.orchestra.run.smithi008.stderr:Setting orchestrator backend to cephadm... 2024-02-10T06:37:03.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:02 smithi008 ceph-mon[24646]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T06:37:03.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:02 smithi008 ceph-mon[24646]: mgrmap e7: smithi008.pxifbt(active, since 1.05487s) 2024-02-10T06:37:03.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:02 smithi008 ceph-mon[24646]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T06:37:04.113 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:03 smithi008 ceph-mon[24646]: mgrmap e8: smithi008.pxifbt(active, since 2s) 2024-02-10T06:37:04.113 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:03 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:04.113 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:03 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:37:04.113 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:03 smithi008 ceph-mon[24646]: [10/Feb/2024:06:37:03] ENGINE Bus STARTING 2024-02-10T06:37:04.113 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:03 smithi008 ceph-mon[24646]: [10/Feb/2024:06:37:03] ENGINE Serving on https://172.21.15.8:7150 2024-02-10T06:37:04.113 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:03 smithi008 ceph-mon[24646]: [10/Feb/2024:06:37:03] ENGINE Bus STARTED 2024-02-10T06:37:04.113 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:03 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:04.113 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:03 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:37:04.113 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: value unchanged 2024-02-10T06:37:04.708 INFO:teuthology.orchestra.run.smithi008.stderr:Generating ssh key... 2024-02-10T06:37:05.017 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:04 smithi008 ceph-mon[24646]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:06.086 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:05 smithi008 ceph-mon[24646]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:06.086 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:05 smithi008 ceph-mon[24646]: Generating ssh key... 2024-02-10T06:37:06.086 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:05 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:06.086 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:05 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:06.906 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD/jxXyowpw4HTHciS2bHYxfetWu3A5XdAvo28U7Tnqz5iLqJm9lcvcOlK9voyiyr8ienlIGIk/ymwRZ/9Ldsm6f9NYpL9lV7yVw4aU+Tij27cmjqVdtuEKSR1ft3oWTiAEaOnPVtune5RLnh0J9S3groH8kDJOpT/ULvcd6jT9DRrYB8J8jZH2EdkALn+NE1hhPHZOW0UBnjNqhKQczpPMbPmwBW7TaDoJakYQUDs8tBqDNl2lxT5KjtZTdfmJsUljiqap3DlKAz3+2eOaG5nvhaz1wviZ21ZRBBHjd01SjsUOUbRb351tfHt9eqUQ73po5UC/qKSk/imXjS05hUxIe6K9FqfXqtCf39qUQ+q2B2dcueUbbcGsXDnHvr8bZ8EBQ5JpUvPlMZU9ZM+JRKPJRUQtRDO5bia+7WgxFgbrXB84rtQe9dzk57HnkSMbxn5nOMVPGQpslzsTYB6sRKu/CCGrPni+Yi14nSp3V3KA3VE34DnAwv673q4bSmYm8U8= ceph-a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:37:07.446 INFO:teuthology.orchestra.run.smithi008.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-10T06:37:07.446 INFO:teuthology.orchestra.run.smithi008.stderr:Adding key to root@localhost authorized_keys... 2024-02-10T06:37:07.446 INFO:teuthology.orchestra.run.smithi008.stderr:Adding host smithi008... 2024-02-10T06:37:08.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:07 smithi008 ceph-mon[24646]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:09.069 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Added host 'smithi008' with addr '172.21.15.8' 2024-02-10T06:37:09.686 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying mon service with default placement... 2024-02-10T06:37:10.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:09 smithi008 ceph-mon[24646]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi008", "addr": "172.21.15.8", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:10.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:09 smithi008 ceph-mon[24646]: Deploying cephadm binary to smithi008 2024-02-10T06:37:10.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:09 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:10.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:09 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:37:10.315 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-10T06:37:11.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:10 smithi008 ceph-mon[24646]: Added host smithi008 2024-02-10T06:37:11.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:10 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:11.137 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying mgr service with default placement... 2024-02-10T06:37:11.803 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-10T06:37:12.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:11 smithi008 ceph-mon[24646]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:12.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:11 smithi008 ceph-mon[24646]: Saving service mon spec with placement count:5 2024-02-10T06:37:12.513 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying crash service with default placement... 2024-02-10T06:37:12.930 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:12 smithi008 ceph-mon[24646]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:12.930 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:12 smithi008 ceph-mon[24646]: Saving service mgr spec with placement count:2 2024-02-10T06:37:12.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:12 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:13.134 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-10T06:37:14.173 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:14 smithi008 ceph-mon[24646]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:14.173 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:14 smithi008 ceph-mon[24646]: Saving service crash spec with placement * 2024-02-10T06:37:14.173 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:14 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:14.173 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:14 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:14.173 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:14 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:14.173 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying prometheus service with default placement... 2024-02-10T06:37:14.790 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-10T06:37:15.464 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying grafana service with default placement... 2024-02-10T06:37:16.040 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:15 smithi008 ceph-mon[24646]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:16.040 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:15 smithi008 ceph-mon[24646]: Saving service prometheus spec with placement count:1 2024-02-10T06:37:16.040 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:15 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:16.572 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-10T06:37:17.124 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying node-exporter service with default placement... 2024-02-10T06:37:17.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:16 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:17.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:16 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:17.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:16 smithi008 ceph-mon[24646]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:17.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:16 smithi008 ceph-mon[24646]: Saving service grafana spec with placement count:1 2024-02-10T06:37:17.238 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:16 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:18.397 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-10T06:37:19.269 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying alertmanager service with default placement... 2024-02-10T06:37:19.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:19 smithi008 ceph-mon[24646]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:19.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:19 smithi008 ceph-mon[24646]: Saving service node-exporter spec with placement * 2024-02-10T06:37:19.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:19 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:19.968 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-10T06:37:21.276 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:20 smithi008 ceph-mon[24646]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:21.276 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:20 smithi008 ceph-mon[24646]: Saving service alertmanager spec with placement count:1 2024-02-10T06:37:21.276 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:20 smithi008 ceph-mon[24646]: from='mgr.14118 172.21.15.8:0/2522108565' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:22.479 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:22 smithi008 ceph-mon[24646]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:22.479 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:22 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/3191138349' entity='client.admin' 2024-02-10T06:37:23.124 INFO:teuthology.orchestra.run.smithi008.stderr:Enabling the dashboard module... 2024-02-10T06:37:23.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:23 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/653304924' entity='client.admin' 2024-02-10T06:37:25.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:24 smithi008 ceph-mon[24646]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:25.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:24 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1937016590' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-10T06:37:25.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:25 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1937016590' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-10T06:37:25.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:25 smithi008 ceph-mon[24646]: mgrmap e9: smithi008.pxifbt(active, since 23s) 2024-02-10T06:37:26.039 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-02-10T06:37:26.039 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 9, 2024-02-10T06:37:26.039 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": true, 2024-02-10T06:37:26.039 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "active_name": "smithi008.pxifbt", 2024-02-10T06:37:26.039 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-10T06:37:26.039 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-02-10T06:37:26.738 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for the mgr to restart... 2024-02-10T06:37:26.738 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mgr epoch 9... 2024-02-10T06:37:26.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:26 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/3772531760' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T06:37:31.818 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:31 smithi008 ceph-mon[24646]: Active manager daemon smithi008.pxifbt restarted 2024-02-10T06:37:31.818 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:31 smithi008 ceph-mon[24646]: Activating manager daemon smithi008.pxifbt 2024-02-10T06:37:31.818 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:31 smithi008 ceph-mon[24646]: osdmap e3: 0 total, 0 up, 0 in 2024-02-10T06:37:31.818 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:31 smithi008 ceph-mon[24646]: mgrmap e10: smithi008.pxifbt(active, starting, since 0.0524651s) 2024-02-10T06:37:31.818 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:31 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:37:31.819 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:31 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr metadata", "who": "smithi008.pxifbt", "id": "smithi008.pxifbt"}]: dispatch 2024-02-10T06:37:31.819 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:31 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:37:31.819 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:31 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:37:31.819 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:31 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:37:31.819 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:31 smithi008 ceph-mon[24646]: Manager daemon smithi008.pxifbt is now available 2024-02-10T06:37:31.819 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:31 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:37:31.819 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:31 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:37:31.819 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:31 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:37:31.819 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:31 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:37:31.820 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-02-10T06:37:31.820 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-02-10T06:37:31.820 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "initialized": true 2024-02-10T06:37:31.820 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-02-10T06:37:32.540 INFO:teuthology.orchestra.run.smithi008.stderr:mgr epoch 9 is available 2024-02-10T06:37:32.541 INFO:teuthology.orchestra.run.smithi008.stderr:Generating a dashboard self-signed certificate... 2024-02-10T06:37:32.820 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:32 smithi008 ceph-mon[24646]: [10/Feb/2024:06:37:31] ENGINE Bus STARTING 2024-02-10T06:37:32.820 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:32 smithi008 ceph-mon[24646]: mgrmap e11: smithi008.pxifbt(active, since 1.05494s) 2024-02-10T06:37:32.820 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:32 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:33.283 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-10T06:37:33.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:33 smithi008 ceph-mon[24646]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T06:37:33.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:33 smithi008 ceph-mon[24646]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T06:37:33.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:33 smithi008 ceph-mon[24646]: [10/Feb/2024:06:37:31] ENGINE Serving on https://172.21.15.8:7150 2024-02-10T06:37:33.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:33 smithi008 ceph-mon[24646]: [10/Feb/2024:06:37:31] ENGINE Bus STARTED 2024-02-10T06:37:33.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:33 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:33.962 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:33 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:33.962 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:33 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:33.962 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:33 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:33.963 INFO:teuthology.orchestra.run.smithi008.stderr:Creating initial admin user... 2024-02-10T06:37:34.922 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:34 smithi008 ceph-mon[24646]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:34.922 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:34 smithi008 ceph-mon[24646]: mgrmap e12: smithi008.pxifbt(active, since 3s) 2024-02-10T06:37:34.922 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$fVhclHaUaPqxwLi3y1uhneBINVMDRsvjwMRk3B8ZoRewf4EtOk652", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707547054, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-10T06:37:35.750 INFO:teuthology.orchestra.run.smithi008.stderr:Fetching dashboard port number... 2024-02-10T06:37:36.011 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:35 smithi008 ceph-mon[24646]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:36.011 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:35 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:36.402 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 8443 2024-02-10T06:37:36.940 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:36 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1430144640' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-10T06:37:37.245 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: disabled 2024-02-10T06:37:37.253 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-02-10T06:37:37.254 INFO:teuthology.orchestra.run.smithi008.stderr:firewalld.service is not enabled 2024-02-10T06:37:37.254 INFO:teuthology.orchestra.run.smithi008.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-10T06:37:37.256 INFO:teuthology.orchestra.run.smithi008.stderr:Ceph Dashboard is now available at: 2024-02-10T06:37:37.256 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-02-10T06:37:37.256 INFO:teuthology.orchestra.run.smithi008.stderr: URL: https://smithi008.front.sepia.ceph.com:8443/ 2024-02-10T06:37:37.256 INFO:teuthology.orchestra.run.smithi008.stderr: User: admin 2024-02-10T06:37:37.256 INFO:teuthology.orchestra.run.smithi008.stderr: Password: 94nonakcn6 2024-02-10T06:37:37.256 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-02-10T06:37:37.256 INFO:teuthology.orchestra.run.smithi008.stderr:Enabling autotune for osd_memory_target 2024-02-10T06:37:39.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:38 smithi008 ceph-mon[24646]: mgrmap e13: smithi008.pxifbt(active, since 7s) 2024-02-10T06:37:39.263 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-02-10T06:37:39.840 INFO:teuthology.orchestra.run.smithi008.stderr:You can access the Ceph CLI with: 2024-02-10T06:37:39.840 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-02-10T06:37:39.840 INFO:teuthology.orchestra.run.smithi008.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:37:39.841 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-02-10T06:37:39.841 INFO:teuthology.orchestra.run.smithi008.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-10T06:37:39.841 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-02-10T06:37:39.841 INFO:teuthology.orchestra.run.smithi008.stderr: ceph telemetry on 2024-02-10T06:37:39.841 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-02-10T06:37:39.841 INFO:teuthology.orchestra.run.smithi008.stderr:For more information see: 2024-02-10T06:37:39.841 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-02-10T06:37:39.841 INFO:teuthology.orchestra.run.smithi008.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-10T06:37:39.841 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-02-10T06:37:39.841 INFO:teuthology.orchestra.run.smithi008.stderr:Bootstrap complete. 2024-02-10T06:37:39.880 INFO:tasks.cephadm:Fetching config... 2024-02-10T06:37:39.880 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T06:37:39.880 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-10T06:37:39.896 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-10T06:37:39.896 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T06:37:39.896 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-10T06:37:39.951 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-10T06:37:39.951 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T06:37:39.952 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/keyring of=/dev/stdout 2024-02-10T06:37:40.019 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-10T06:37:40.019 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T06:37:40.020 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-10T06:37:40.073 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-10T06:37:40.074 DEBUG:teuthology.orchestra.run.smithi008:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD/jxXyowpw4HTHciS2bHYxfetWu3A5XdAvo28U7Tnqz5iLqJm9lcvcOlK9voyiyr8ienlIGIk/ymwRZ/9Ldsm6f9NYpL9lV7yVw4aU+Tij27cmjqVdtuEKSR1ft3oWTiAEaOnPVtune5RLnh0J9S3groH8kDJOpT/ULvcd6jT9DRrYB8J8jZH2EdkALn+NE1hhPHZOW0UBnjNqhKQczpPMbPmwBW7TaDoJakYQUDs8tBqDNl2lxT5KjtZTdfmJsUljiqap3DlKAz3+2eOaG5nvhaz1wviZ21ZRBBHjd01SjsUOUbRb351tfHt9eqUQ73po5UC/qKSk/imXjS05hUxIe6K9FqfXqtCf39qUQ+q2B2dcueUbbcGsXDnHvr8bZ8EBQ5JpUvPlMZU9ZM+JRKPJRUQtRDO5bia+7WgxFgbrXB84rtQe9dzk57HnkSMbxn5nOMVPGQpslzsTYB6sRKu/CCGrPni+Yi14nSp3V3KA3VE34DnAwv673q4bSmYm8U8= ceph-a5f941ec-c7de-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T06:37:40.152 INFO:teuthology.orchestra.run.smithi008.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD/jxXyowpw4HTHciS2bHYxfetWu3A5XdAvo28U7Tnqz5iLqJm9lcvcOlK9voyiyr8ienlIGIk/ymwRZ/9Ldsm6f9NYpL9lV7yVw4aU+Tij27cmjqVdtuEKSR1ft3oWTiAEaOnPVtune5RLnh0J9S3groH8kDJOpT/ULvcd6jT9DRrYB8J8jZH2EdkALn+NE1hhPHZOW0UBnjNqhKQczpPMbPmwBW7TaDoJakYQUDs8tBqDNl2lxT5KjtZTdfmJsUljiqap3DlKAz3+2eOaG5nvhaz1wviZ21ZRBBHjd01SjsUOUbRb351tfHt9eqUQ73po5UC/qKSk/imXjS05hUxIe6K9FqfXqtCf39qUQ+q2B2dcueUbbcGsXDnHvr8bZ8EBQ5JpUvPlMZU9ZM+JRKPJRUQtRDO5bia+7WgxFgbrXB84rtQe9dzk57HnkSMbxn5nOMVPGQpslzsTYB6sRKu/CCGrPni+Yi14nSp3V3KA3VE34DnAwv673q4bSmYm8U8= ceph-a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:37:40.168 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD/jxXyowpw4HTHciS2bHYxfetWu3A5XdAvo28U7Tnqz5iLqJm9lcvcOlK9voyiyr8ienlIGIk/ymwRZ/9Ldsm6f9NYpL9lV7yVw4aU+Tij27cmjqVdtuEKSR1ft3oWTiAEaOnPVtune5RLnh0J9S3groH8kDJOpT/ULvcd6jT9DRrYB8J8jZH2EdkALn+NE1hhPHZOW0UBnjNqhKQczpPMbPmwBW7TaDoJakYQUDs8tBqDNl2lxT5KjtZTdfmJsUljiqap3DlKAz3+2eOaG5nvhaz1wviZ21ZRBBHjd01SjsUOUbRb351tfHt9eqUQ73po5UC/qKSk/imXjS05hUxIe6K9FqfXqtCf39qUQ+q2B2dcueUbbcGsXDnHvr8bZ8EBQ5JpUvPlMZU9ZM+JRKPJRUQtRDO5bia+7WgxFgbrXB84rtQe9dzk57HnkSMbxn5nOMVPGQpslzsTYB6sRKu/CCGrPni+Yi14nSp3V3KA3VE34DnAwv673q4bSmYm8U8= ceph-a5f941ec-c7de-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T06:37:40.215 INFO:teuthology.orchestra.run.smithi029.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD/jxXyowpw4HTHciS2bHYxfetWu3A5XdAvo28U7Tnqz5iLqJm9lcvcOlK9voyiyr8ienlIGIk/ymwRZ/9Ldsm6f9NYpL9lV7yVw4aU+Tij27cmjqVdtuEKSR1ft3oWTiAEaOnPVtune5RLnh0J9S3groH8kDJOpT/ULvcd6jT9DRrYB8J8jZH2EdkALn+NE1hhPHZOW0UBnjNqhKQczpPMbPmwBW7TaDoJakYQUDs8tBqDNl2lxT5KjtZTdfmJsUljiqap3DlKAz3+2eOaG5nvhaz1wviZ21ZRBBHjd01SjsUOUbRb351tfHt9eqUQ73po5UC/qKSk/imXjS05hUxIe6K9FqfXqtCf39qUQ+q2B2dcueUbbcGsXDnHvr8bZ8EBQ5JpUvPlMZU9ZM+JRKPJRUQtRDO5bia+7WgxFgbrXB84rtQe9dzk57HnkSMbxn5nOMVPGQpslzsTYB6sRKu/CCGrPni+Yi14nSp3V3KA3VE34DnAwv673q4bSmYm8U8= ceph-a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:37:40.231 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-10T06:37:40.400 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:40 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/465010487' entity='client.admin' 2024-02-10T06:37:43.887 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-10T06:37:43.887 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-10T06:37:44.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:44 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/51727412' entity='client.admin' 2024-02-10T06:37:46.737 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi029 2024-02-10T06:37:46.737 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T06:37:46.737 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/etc/ceph/ceph.conf 2024-02-10T06:37:46.755 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T06:37:46.755 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:37:46.813 INFO:tasks.cephadm:Adding host smithi029 to orchestrator... 2024-02-10T06:37:46.813 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch host add smithi029 2024-02-10T06:37:47.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:46 smithi008 ceph-mon[24646]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:47.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:46 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:47.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:46 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:47.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:46 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd/host:smithi008", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:37:47.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:46 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:48.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:47 smithi008 ceph-mon[24646]: Deploying daemon alertmanager.smithi008 on smithi008 2024-02-10T06:37:50.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:49 smithi008 ceph-mon[24646]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi029", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:37:50.525 INFO:teuthology.orchestra.run.smithi008.stdout:Added host 'smithi029' with addr '172.21.15.29' 2024-02-10T06:37:51.709 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-10T06:37:51.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:51 smithi008 ceph-mon[24646]: Deploying cephadm binary to smithi029 2024-02-10T06:37:51.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:51 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:51.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:51 smithi008 ceph-mon[24646]: Added host smithi029 2024-02-10T06:37:52.746 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:52 smithi008 ceph-mon[24646]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:55.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:54 smithi008 ceph-mon[24646]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:56.250 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:37:56.251 INFO:teuthology.orchestra.run.smithi008.stdout:[{"addr": "172.21.15.8", "hostname": "smithi008", "labels": [], "status": ""}, {"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}] 2024-02-10T06:37:57.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:56 smithi008 ceph-mon[24646]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:57.272 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-10T06:37:57.272 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph osd crush tunables default 2024-02-10T06:37:57.994 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:57 smithi008 ceph-mon[24646]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:37:57.994 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:57 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:37:57.995 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:57 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:37:57.995 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:57 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-10T06:37:57.995 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:57 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:37:59.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:58 smithi008 ceph-mon[24646]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:37:59.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:37:58 smithi008 ceph-mon[24646]: Deploying daemon crash.smithi008 on smithi008 2024-02-10T06:38:00.751 INFO:teuthology.orchestra.run.smithi008.stderr:adjusted tunables profile to default 2024-02-10T06:38:00.894 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:00 smithi008 ceph-mon[24646]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:00.894 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:00 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1354366183' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T06:38:01.294 INFO:tasks.cephadm:Adding mon.smithi008 on smithi008 2024-02-10T06:38:01.294 INFO:tasks.cephadm:Adding mon.smithi029 on smithi029 2024-02-10T06:38:01.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch apply mon '2;smithi008:172.21.15.8=smithi008;smithi029:172.21.15.29=smithi029' 2024-02-10T06:38:02.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:01 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1354366183' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T06:38:02.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:01 smithi008 ceph-mon[24646]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T06:38:02.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:01 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:02.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:01 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:01 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:01 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:38:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:01 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:02.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:01 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:03.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:02 smithi008 ceph-mon[24646]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:03.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:02 smithi008 ceph-mon[24646]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:38:03.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:02 smithi008 ceph-mon[24646]: Deploying daemon grafana.smithi008 on smithi008 2024-02-10T06:38:03.111 INFO:teuthology.orchestra.run.smithi029.stdout:Scheduled mon update... 2024-02-10T06:38:03.606 DEBUG:teuthology.orchestra.run.smithi029:mon.smithi029> sudo journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mon.smithi029.service 2024-02-10T06:38:03.609 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:38:03.609 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:38:04.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:04 smithi008 ceph-mon[24646]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:04.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:04 smithi008 ceph-mon[24646]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi008:172.21.15.8=smithi008;smithi029:172.21.15.29=smithi029", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:38:04.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:04 smithi008 ceph-mon[24646]: Saving service mon spec with placement smithi008:172.21.15.8=smithi008;smithi029:172.21.15.29=smithi029;count:2 2024-02-10T06:38:04.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:04 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:05.567 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:38:05.567 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":1,"fsid":"a5f941ec-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:36:26.499501Z","created":"2024-02-10T06:36:26.499501Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:38:05.569 INFO:teuthology.orchestra.run.smithi029.stderr:dumped monmap epoch 1 2024-02-10T06:38:06.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:06 smithi008 ceph-mon[24646]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:06.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:06 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/208466498' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:38:07.119 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:38:07.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:38:08.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:08 smithi008 ceph-mon[24646]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:08.981 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:38:08.981 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":1,"fsid":"a5f941ec-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:36:26.499501Z","created":"2024-02-10T06:36:26.499501Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:38:08.983 INFO:teuthology.orchestra.run.smithi029.stderr:dumped monmap epoch 1 2024-02-10T06:38:10.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:09 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/3791679815' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:38:10.535 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:38:10.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:38:11.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:10 smithi008 ceph-mon[24646]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:12.382 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:38:12.383 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":1,"fsid":"a5f941ec-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:36:26.499501Z","created":"2024-02-10T06:36:26.499501Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:38:12.383 INFO:teuthology.orchestra.run.smithi029.stderr:dumped monmap epoch 1 2024-02-10T06:38:13.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:12 smithi008 ceph-mon[24646]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:13.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:12 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/2426116347' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:38:14.145 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:38:14.145 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:38:14.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:14 smithi008 ceph-mon[24646]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:14.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:14 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:14.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:14 smithi008 ceph-mon[24646]: Deploying daemon node-exporter.smithi008 on smithi008 2024-02-10T06:38:15.998 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:38:15.999 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":1,"fsid":"a5f941ec-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:36:26.499501Z","created":"2024-02-10T06:36:26.499501Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:38:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:dumped monmap epoch 1 2024-02-10T06:38:16.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:16 smithi008 ceph-mon[24646]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:16.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:16 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:16.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:16 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/251535537' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:38:17.575 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:38:17.576 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:38:18.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:17 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:18.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:17 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T06:38:18.971 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:18 smithi008 ceph-mon[24646]: from='mgr.14160 172.21.15.8:0/1380987492' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T06:38:18.971 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:18 smithi008 ceph-mon[24646]: mgrmap e14: smithi008.pxifbt(active, since 46s) 2024-02-10T06:38:19.430 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:38:19.430 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":1,"fsid":"a5f941ec-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:36:26.499501Z","created":"2024-02-10T06:36:26.499501Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:38:19.432 INFO:teuthology.orchestra.run.smithi029.stderr:dumped monmap epoch 1 2024-02-10T06:38:20.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:19 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/3683343187' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:38:20.984 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:38:20.985 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:38:22.801 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:38:22.801 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":1,"fsid":"a5f941ec-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:36:26.499501Z","created":"2024-02-10T06:36:26.499501Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:38:22.803 INFO:teuthology.orchestra.run.smithi029.stderr:dumped monmap epoch 1 2024-02-10T06:38:23.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:22 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/653003431' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:38:24.369 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:38:24.369 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:38:24.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: Active manager daemon smithi008.pxifbt restarted 2024-02-10T06:38:24.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: Activating manager daemon smithi008.pxifbt 2024-02-10T06:38:24.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: osdmap e5: 0 total, 0 up, 0 in 2024-02-10T06:38:24.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: mgrmap e15: smithi008.pxifbt(active, starting, since 0.21627s) 2024-02-10T06:38:24.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:38:24.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr metadata", "who": "smithi008.pxifbt", "id": "smithi008.pxifbt"}]: dispatch 2024-02-10T06:38:24.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:38:24.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:38:24.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:38:24.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: Manager daemon smithi008.pxifbt is now available 2024-02-10T06:38:24.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:38:24.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:38:24.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:24.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:38:24.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:24.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:38:24.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:38:26.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:25 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:26.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:25 smithi008 ceph-mon[24646]: mgrmap e16: smithi008.pxifbt(active, since 1.21965s) 2024-02-10T06:38:26.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:25 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:26.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:25 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:26.234 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:38:26.234 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":1,"fsid":"a5f941ec-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:36:26.499501Z","created":"2024-02-10T06:36:26.499501Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:38:26.236 INFO:teuthology.orchestra.run.smithi029.stderr:dumped monmap epoch 1 2024-02-10T06:38:27.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:26 smithi008 ceph-mon[24646]: [10/Feb/2024:06:38:25] ENGINE Bus STARTING 2024-02-10T06:38:27.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:26 smithi008 ceph-mon[24646]: [10/Feb/2024:06:38:25] ENGINE Serving on https://172.21.15.8:7150 2024-02-10T06:38:27.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:26 smithi008 ceph-mon[24646]: [10/Feb/2024:06:38:25] ENGINE Bus STARTED 2024-02-10T06:38:27.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:26 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:27.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:26 smithi008 ceph-mon[24646]: mgrmap e17: smithi008.pxifbt(active, since 2s) 2024-02-10T06:38:27.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:26 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/169538630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:38:27.866 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:38:27.866 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:38:30.476 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:38:30.476 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":1,"fsid":"a5f941ec-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:36:26.499501Z","created":"2024-02-10T06:36:26.499501Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:38:30.478 INFO:teuthology.orchestra.run.smithi029.stderr:dumped monmap epoch 1 2024-02-10T06:38:31.112 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:30 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:31.113 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:30 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:31.113 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:30 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:38:31.113 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:30 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/3008511147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:38:32.026 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:38:32.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:38:32.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:32 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:32.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:32 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:32.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:32 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd/host:smithi008", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:38:32.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:32 smithi008 ceph-mon[24646]: Updating smithi008:/etc/ceph/ceph.conf 2024-02-10T06:38:33.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:33 smithi008 ceph-mon[24646]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:38:33.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:33 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:33.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:33 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:33.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:33 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:38:33.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:33 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-10T06:38:33.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:33 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:33.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:33 smithi008 ceph-mon[24646]: Deploying daemon crash.smithi029 on smithi029 2024-02-10T06:38:33.873 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:38:33.874 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":1,"fsid":"a5f941ec-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:36:26.499501Z","created":"2024-02-10T06:36:26.499501Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:38:33.876 INFO:teuthology.orchestra.run.smithi029.stderr:dumped monmap epoch 1 2024-02-10T06:38:35.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:34 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/2036367247' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:38:35.461 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:38:35.461 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:38:37.496 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:38:37.496 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":1,"fsid":"a5f941ec-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:36:26.499501Z","created":"2024-02-10T06:36:26.499501Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:38:37.498 INFO:teuthology.orchestra.run.smithi029.stderr:dumped monmap epoch 1 2024-02-10T06:38:38.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:37 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/858034018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:38:39.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:38 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:39.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:38 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi029.fuvrtf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:38:39.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:38 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi029.fuvrtf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T06:38:39.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:38 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:38:39.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:38 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:39.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:38 smithi008 ceph-mon[24646]: Deploying daemon mgr.smithi029.fuvrtf on smithi029 2024-02-10T06:38:39.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:38 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:39.432 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:38:39.432 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:38:41.242 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:38:41.242 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":1,"fsid":"a5f941ec-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:36:26.499501Z","created":"2024-02-10T06:36:26.499501Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:38:41.243 INFO:teuthology.orchestra.run.smithi029.stderr:dumped monmap epoch 1 2024-02-10T06:38:41.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:41 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/1761816092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:38:42.833 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:38:42.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:38:43.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:43 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:43.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:43 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:38:43.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:43 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:43.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:43 smithi008 ceph-mon[24646]: Deploying daemon mon.smithi029 on smithi029 2024-02-10T06:38:45.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:44 smithi008 ceph-mon[24646]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:45.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:46.366 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:38:46.366 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":1,"fsid":"a5f941ec-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:36:26.499501Z","created":"2024-02-10T06:36:26.499501Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:38:46.368 INFO:teuthology.orchestra.run.smithi029.stderr:dumped monmap epoch 1 2024-02-10T06:38:47.280 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:47 smithi029 ceph-mon[27077]: mon.smithi029@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-02-10T06:38:47.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:46 smithi008 ceph-mon[24646]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:47.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:46 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/1810210451' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:38:47.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:47 smithi029 bash[27036]: a790a61584bfb9527cac4501975fbd9af046e5da883706ed4b1c7f2eae2fa2ef 2024-02-10T06:38:47.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:47 smithi029 systemd[1]: Started Ceph mon.smithi029 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:38:48.009 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:38:48.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:38:52.590 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:38:52.590 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: mon.smithi008 calling monitor election 2024-02-10T06:38:52.590 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:38:52.590 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/3218105467' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/crt"}]: dispatch 2024-02-10T06:38:52.590 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:52.590 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:38:52.590 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:38:52.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: mon.smithi029 calling monitor election 2024-02-10T06:38:52.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:52.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:38:52.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:38:52.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:38:52.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: mon.smithi008 is new leader, mons smithi008,smithi029 in quorum (ranks 0,1) 2024-02-10T06:38:52.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: monmap e2: 2 mons at {smithi008=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0],smithi029=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} 2024-02-10T06:38:52.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: fsmap 2024-02-10T06:38:52.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: osdmap e5: 0 total, 0 up, 0 in 2024-02-10T06:38:52.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: mgrmap e17: smithi008.pxifbt(active, since 28s) 2024-02-10T06:38:52.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: Standby manager daemon smithi029.fuvrtf started 2024-02-10T06:38:52.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/3218105467' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:38:52.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: overall HEALTH_OK 2024-02-10T06:38:52.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/3218105467' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/key"}]: dispatch 2024-02-10T06:38:52.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/3218105467' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:38:52.591 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:52 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:52.688 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:38:52.688 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: mon.smithi008 calling monitor election 2024-02-10T06:38:52.688 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:38:52.689 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.29:0/3218105467' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/crt"}]: dispatch 2024-02-10T06:38:52.689 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:52.689 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:38:52.689 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:38:52.689 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: mon.smithi029 calling monitor election 2024-02-10T06:38:52.689 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:52.689 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:38:52.689 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:38:52.689 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:38:52.689 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: mon.smithi008 is new leader, mons smithi008,smithi029 in quorum (ranks 0,1) 2024-02-10T06:38:52.689 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: monmap e2: 2 mons at {smithi008=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0],smithi029=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} 2024-02-10T06:38:52.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: fsmap 2024-02-10T06:38:52.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: osdmap e5: 0 total, 0 up, 0 in 2024-02-10T06:38:52.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: mgrmap e17: smithi008.pxifbt(active, since 28s) 2024-02-10T06:38:52.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: Standby manager daemon smithi029.fuvrtf started 2024-02-10T06:38:52.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.29:0/3218105467' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:38:52.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: overall HEALTH_OK 2024-02-10T06:38:52.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.29:0/3218105467' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/key"}]: dispatch 2024-02-10T06:38:52.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.29:0/3218105467' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:38:52.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:52 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:53.033 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T06:38:53.034 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":2,"fsid":"a5f941ec-c7de-11ee-95b8-87774f69a715","modified":"2024-02-10T06:38:47.232235Z","created":"2024-02-10T06:36:26.499501Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T06:38:53.035 INFO:teuthology.orchestra.run.smithi029.stderr:dumped monmap epoch 2 2024-02-10T06:38:53.619 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:53 smithi029 ceph-mon[27077]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:53.619 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:53 smithi029 ceph-mon[27077]: Deploying daemon node-exporter.smithi029 on smithi029 2024-02-10T06:38:53.619 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:53 smithi029 ceph-mon[27077]: mgrmap e18: smithi008.pxifbt(active, since 28s), standbys: smithi029.fuvrtf 2024-02-10T06:38:53.619 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:53 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr metadata", "who": "smithi029.fuvrtf", "id": "smithi029.fuvrtf"}]: dispatch 2024-02-10T06:38:53.619 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:53 smithi029 ceph-mon[27077]: from='client.? 172.21.15.29:0/1023479335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:38:53.619 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:53 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:38:53.642 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-10T06:38:53.642 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph config generate-minimal-conf 2024-02-10T06:38:53.667 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:53 smithi008 ceph-mon[24646]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:53.667 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:53 smithi008 ceph-mon[24646]: Deploying daemon node-exporter.smithi029 on smithi029 2024-02-10T06:38:53.667 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:53 smithi008 ceph-mon[24646]: mgrmap e18: smithi008.pxifbt(active, since 28s), standbys: smithi029.fuvrtf 2024-02-10T06:38:53.667 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:53 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr metadata", "who": "smithi029.fuvrtf", "id": "smithi029.fuvrtf"}]: dispatch 2024-02-10T06:38:53.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:53 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/1023479335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:38:53.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:53 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:38:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:54 smithi029 ceph-mon[27077]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:54 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:55.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:54 smithi008 ceph-mon[24646]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:55.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:54 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:55.651 INFO:teuthology.orchestra.run.smithi008.stdout:# minimal ceph.conf for a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:38:55.652 INFO:teuthology.orchestra.run.smithi008.stdout:[global] 2024-02-10T06:38:55.652 INFO:teuthology.orchestra.run.smithi008.stdout: fsid = a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:38:55.652 INFO:teuthology.orchestra.run.smithi008.stdout: mon_host = [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2024-02-10T06:38:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:55 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1019301822' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:56.268 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:55 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1019301822' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:38:56.291 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-10T06:38:56.291 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T06:38:56.292 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T06:38:56.360 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T06:38:56.360 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:38:56.427 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T06:38:56.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T06:38:56.462 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T06:38:56.462 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:38:56.531 INFO:tasks.cephadm:Deploying OSDs... 2024-02-10T06:38:56.532 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T06:38:56.532 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T06:38:56.547 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T06:38:56.547 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_1 2024-02-10T06:38:56.601 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T06:38:56.601 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:38:56.601 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-10T06:38:56.601 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:38:56.601 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:38:56.601 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-02-10 06:38:14.050977863 +0000 2024-02-10T06:38:56.601 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-02-10 06:35:26.442332743 +0000 2024-02-10T06:38:56.601 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-02-10 06:35:26.442332743 +0000 2024-02-10T06:38:56.602 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-02-10 06:35:26.442332743 +0000 2024-02-10T06:38:56.602 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T06:38:56.665 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-02-10T06:38:56.665 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-02-10T06:38:56.665 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000107011 s, 4.8 MB/s 2024-02-10T06:38:56.667 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T06:38:56.722 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_2 2024-02-10T06:38:56.777 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T06:38:56.778 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:38:56.778 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-02-10T06:38:56.778 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:38:56.778 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:38:56.778 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-02-10 06:38:14.050977863 +0000 2024-02-10T06:38:56.778 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-02-10 06:35:26.769329117 +0000 2024-02-10T06:38:56.778 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-02-10 06:35:26.769329117 +0000 2024-02-10T06:38:56.778 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-02-10 06:35:26.769329117 +0000 2024-02-10T06:38:56.778 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T06:38:56.844 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-02-10T06:38:56.845 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-02-10T06:38:56.845 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 8.9163e-05 s, 5.7 MB/s 2024-02-10T06:38:56.845 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T06:38:56.900 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_3 2024-02-10T06:38:56.954 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T06:38:56.954 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:38:56.954 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-02-10T06:38:56.954 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:38:56.954 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:38:56.954 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-02-10 06:38:14.050977863 +0000 2024-02-10T06:38:56.954 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-02-10 06:35:27.099325457 +0000 2024-02-10T06:38:56.954 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-02-10 06:35:27.099325457 +0000 2024-02-10T06:38:56.954 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-02-10 06:35:27.099325457 +0000 2024-02-10T06:38:56.955 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T06:38:57.019 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-02-10T06:38:57.019 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-02-10T06:38:57.019 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000196071 s, 2.6 MB/s 2024-02-10T06:38:57.021 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T06:38:57.076 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_4 2024-02-10T06:38:57.131 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T06:38:57.131 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:38:57.131 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-10T06:38:57.131 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:38:57.131 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:38:57.132 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-02-10 06:38:14.051977852 +0000 2024-02-10T06:38:57.132 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-02-10 06:35:27.434321742 +0000 2024-02-10T06:38:57.132 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-02-10 06:35:27.434321742 +0000 2024-02-10T06:38:57.132 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-02-10 06:35:27.434321742 +0000 2024-02-10T06:38:57.132 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T06:38:57.196 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-02-10T06:38:57.197 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-02-10T06:38:57.197 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 9.6543e-05 s, 5.3 MB/s 2024-02-10T06:38:57.198 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T06:38:57.253 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T06:38:57.253 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T06:38:57.272 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T06:38:57.272 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-02-10T06:38:57.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:56 smithi008 ceph-mon[24646]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:57.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:56 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:57.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:56 smithi008 ceph-mon[24646]: Deploying daemon prometheus.smithi008 on smithi008 2024-02-10T06:38:57.330 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T06:38:57.330 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:38:57.330 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-10T06:38:57.330 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:38:57.330 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:38:57.331 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-02-10 06:38:52.888445219 +0000 2024-02-10T06:38:57.331 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-02-10 06:35:33.482923803 +0000 2024-02-10T06:38:57.331 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-02-10 06:35:33.482923803 +0000 2024-02-10T06:38:57.331 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-02-10 06:35:33.482923803 +0000 2024-02-10T06:38:57.331 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T06:38:57.397 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:56 smithi029 ceph-mon[27077]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:57.397 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:56 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:57.397 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:56 smithi029 ceph-mon[27077]: Deploying daemon prometheus.smithi008 on smithi008 2024-02-10T06:38:57.400 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-02-10T06:38:57.400 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-02-10T06:38:57.400 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000126258 s, 4.1 MB/s 2024-02-10T06:38:57.402 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T06:38:57.459 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-02-10T06:38:57.516 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T06:38:57.516 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:38:57.516 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-10T06:38:57.516 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:38:57.516 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:38:57.516 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-02-10 06:38:52.888445219 +0000 2024-02-10T06:38:57.516 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-02-10 06:35:33.815918362 +0000 2024-02-10T06:38:57.516 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-02-10 06:35:33.815918362 +0000 2024-02-10T06:38:57.517 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-02-10 06:35:33.815918362 +0000 2024-02-10T06:38:57.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T06:38:57.584 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-02-10T06:38:57.584 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-02-10T06:38:57.585 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.00010263 s, 5.0 MB/s 2024-02-10T06:38:57.586 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T06:38:57.643 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-02-10T06:38:57.699 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T06:38:57.699 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:38:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-02-10T06:38:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:38:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:38:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-02-10 06:38:52.889445203 +0000 2024-02-10T06:38:57.699 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-02-10 06:35:34.138913085 +0000 2024-02-10T06:38:57.700 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-02-10 06:35:34.138913085 +0000 2024-02-10T06:38:57.700 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-02-10 06:35:34.138913085 +0000 2024-02-10T06:38:57.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T06:38:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-02-10T06:38:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-02-10T06:38:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000104991 s, 4.9 MB/s 2024-02-10T06:38:57.767 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T06:38:57.824 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-02-10T06:38:57.880 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T06:38:57.880 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:38:57.880 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-10T06:38:57.880 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:38:57.881 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:38:57.881 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-02-10 06:38:52.889445203 +0000 2024-02-10T06:38:57.881 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-02-10 06:35:34.487907383 +0000 2024-02-10T06:38:57.881 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-02-10 06:35:34.487907383 +0000 2024-02-10T06:38:57.881 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-02-10 06:35:34.487907383 +0000 2024-02-10T06:38:57.881 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T06:38:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-02-10T06:38:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-02-10T06:38:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.7617e-05 s, 5.2 MB/s 2024-02-10T06:38:57.949 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T06:38:58.006 INFO:tasks.cephadm:Deploying osd.0 on smithi008 with /dev/vg_nvme/lv_4... 2024-02-10T06:38:58.006 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T06:38:59.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:59 smithi008 ceph-mon[24646]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:59.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:38:59 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:38:59.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:59 smithi029 ceph-mon[27077]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:38:59.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:38:59 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:01.516 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T06:39:01.517 INFO:teuthology.orchestra.run.smithi008.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T06:39:01.517 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10+0 records in 2024-02-10T06:39:01.517 INFO:teuthology.orchestra.run.smithi008.stderr:10+0 records out 2024-02-10T06:39:01.517 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00890921 s, 1.2 GB/s 2024-02-10T06:39:01.517 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping successful for: 2024-02-10T06:39:01.517 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:01 smithi008 ceph-mon[24646]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:01.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:01 smithi029 ceph-mon[27077]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:02.953 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi008:vg_nvme/lv_4 2024-02-10T06:39:03.520 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:03 smithi008 ceph-mon[24646]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:03.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:03 smithi029 ceph-mon[27077]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:05.008 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:04 smithi008 ceph-mon[24646]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:05.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:04 smithi029 ceph-mon[27077]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:07.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:06 smithi029 ceph-mon[27077]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:07.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:06 smithi029 ceph-mon[27077]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:39:07.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:06 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:39:07.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:06 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:39:07.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:06 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:07.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:06 smithi008 ceph-mon[24646]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:07.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:06 smithi008 ceph-mon[24646]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:39:07.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:06 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:39:07.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:06 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:39:07.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:06 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:09.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:08 smithi008 ceph-mon[24646]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:08 smithi029 ceph-mon[27077]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:10.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:10 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:10.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:10 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:39:10.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:10 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:10.760 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:10 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:39:10.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:10 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:10.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:10 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:39:10.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:10 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:10.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:10 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:39:11.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:11 smithi029 ceph-mon[27077]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:11.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:11 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/682188314' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff16b296-565e-40ff-b128-8f652db427d4"}]: dispatch 2024-02-10T06:39:11.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:11 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/682188314' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff16b296-565e-40ff-b128-8f652db427d4"}]': finished 2024-02-10T06:39:11.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:11 smithi029 ceph-mon[27077]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T06:39:11.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:11 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:39:11.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:11 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:11.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:11 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/4031433390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:39:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:11 smithi008 ceph-mon[24646]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:11 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/682188314' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff16b296-565e-40ff-b128-8f652db427d4"}]: dispatch 2024-02-10T06:39:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:11 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/682188314' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff16b296-565e-40ff-b128-8f652db427d4"}]': finished 2024-02-10T06:39:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:11 smithi008 ceph-mon[24646]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T06:39:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:11 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:39:11.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:11 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:11.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:11 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/4031433390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:39:12.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:12 smithi029 ceph-mon[27077]: Updating smithi029:/etc/ceph/ceph.conf 2024-02-10T06:39:12.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:12 smithi029 ceph-mon[27077]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:39:12.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:12 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:12.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:12 smithi008 ceph-mon[24646]: Updating smithi029:/etc/ceph/ceph.conf 2024-02-10T06:39:12.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:12 smithi008 ceph-mon[24646]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:39:12.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:12 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:13.770 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:13 smithi008 ceph-mon[24646]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:13.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:13 smithi029 ceph-mon[27077]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:15.020 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:14 smithi008 ceph-mon[24646]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:15.020 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:14 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:15.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:14 smithi029 ceph-mon[27077]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:15.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:14 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:16.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:16 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:16.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:16 smithi008 ceph-mon[24646]: Updating smithi008:/etc/ceph/ceph.conf 2024-02-10T06:39:16.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:16 smithi008 ceph-mon[24646]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:16.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:16 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:16.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:16 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:16.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:16 smithi008 ceph-mon[24646]: Reconfiguring mon.smithi008 (unknown last config time)... 2024-02-10T06:39:16.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:16 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:39:16.899 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:16 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:39:16.899 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:16 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:16.899 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:16 smithi008 ceph-mon[24646]: Reconfiguring daemon mon.smithi008 on smithi008 2024-02-10T06:39:17.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:16 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:17.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:16 smithi029 ceph-mon[27077]: Updating smithi008:/etc/ceph/ceph.conf 2024-02-10T06:39:17.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:16 smithi029 ceph-mon[27077]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:17.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:16 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:17.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:16 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:17.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:16 smithi029 ceph-mon[27077]: Reconfiguring mon.smithi008 (unknown last config time)... 2024-02-10T06:39:17.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:16 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:39:17.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:16 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:39:17.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:16 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:17.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:16 smithi029 ceph-mon[27077]: Reconfiguring daemon mon.smithi008 on smithi008 2024-02-10T06:39:19.596 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:19 smithi008 ceph-mon[24646]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:19.596 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:19 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:19.596 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:19 smithi008 ceph-mon[24646]: Reconfiguring mgr.smithi008.pxifbt (unknown last config time)... 2024-02-10T06:39:19.596 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:19 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.pxifbt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:39:19.596 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:19 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:39:19.596 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:19 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:19.596 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:19 smithi008 ceph-mon[24646]: Reconfiguring daemon mgr.smithi008.pxifbt on smithi008 2024-02-10T06:39:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:19 smithi029 ceph-mon[27077]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:19 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:19 smithi029 ceph-mon[27077]: Reconfiguring mgr.smithi008.pxifbt (unknown last config time)... 2024-02-10T06:39:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:19 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.pxifbt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:39:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:19 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:39:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:19 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:19 smithi029 ceph-mon[27077]: Reconfiguring daemon mgr.smithi008.pxifbt on smithi008 2024-02-10T06:39:20.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:20 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T06:39:20.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:20 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:20.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:20 smithi029 ceph-mon[27077]: Deploying daemon osd.0 on smithi008 2024-02-10T06:39:20.761 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:20 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T06:39:20.761 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:20 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:20.762 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:20 smithi008 ceph-mon[24646]: Deploying daemon osd.0 on smithi008 2024-02-10T06:39:21.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:21 smithi008 ceph-mon[24646]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:21.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:21 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:21.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:21 smithi029 ceph-mon[27077]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:21.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:21 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:22.422 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:22 smithi008 ceph-mon[24646]: Reconfiguring alertmanager.smithi008 (dependencies changed)... 2024-02-10T06:39:22.422 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:22 smithi008 ceph-mon[24646]: Reconfiguring daemon alertmanager.smithi008 on smithi008 2024-02-10T06:39:22.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:22 smithi029 ceph-mon[27077]: Reconfiguring alertmanager.smithi008 (dependencies changed)... 2024-02-10T06:39:22.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:22 smithi029 ceph-mon[27077]: Reconfiguring daemon alertmanager.smithi008 on smithi008 2024-02-10T06:39:23.580 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:23 smithi008 ceph-mon[24646]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:23.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:23 smithi029 ceph-mon[27077]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:25.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:24 smithi008 ceph-mon[24646]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:25.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:39:25.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:39:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:24 smithi029 ceph-mon[27077]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:39:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:39:27.276 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:27 smithi008 ceph-mon[24646]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:27.277 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:27 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:27.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:27 smithi029 ceph-mon[27077]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:27.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:27 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:29.235 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:29 smithi008 ceph-mon[24646]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:29.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:29 smithi029 ceph-mon[27077]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:30.125 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:30 smithi008 ceph-mon[24646]: from='osd.0 [v2:172.21.15.8:6802/903496854,v1:172.21.15.8:6803/903496854]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T06:39:30.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:30 smithi029 ceph-mon[27077]: from='osd.0 [v2:172.21.15.8:6802/903496854,v1:172.21.15.8:6803/903496854]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T06:39:31.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:31 smithi008 ceph-mon[24646]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:31.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:31 smithi008 ceph-mon[24646]: from='osd.0 [v2:172.21.15.8:6802/903496854,v1:172.21.15.8:6803/903496854]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T06:39:31.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:31 smithi008 ceph-mon[24646]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T06:39:31.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:31 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:39:31.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:31 smithi008 ceph-mon[24646]: from='osd.0 [v2:172.21.15.8:6802/903496854,v1:172.21.15.8:6803/903496854]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T06:39:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:31 smithi029 ceph-mon[27077]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:31 smithi029 ceph-mon[27077]: from='osd.0 [v2:172.21.15.8:6802/903496854,v1:172.21.15.8:6803/903496854]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T06:39:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:31 smithi029 ceph-mon[27077]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T06:39:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:31 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:39:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:31 smithi029 ceph-mon[27077]: from='osd.0 [v2:172.21.15.8:6802/903496854,v1:172.21.15.8:6803/903496854]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T06:39:32.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:32 smithi008 ceph-mon[24646]: from='osd.0 [v2:172.21.15.8:6802/903496854,v1:172.21.15.8:6803/903496854]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T06:39:32.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:32 smithi008 ceph-mon[24646]: osdmap e8: 1 total, 0 up, 1 in 2024-02-10T06:39:32.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:32 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:39:32.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:32 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:39:32.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:32 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:32.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:32 smithi008 ceph-mon[24646]: Reconfiguring crash.smithi008 (monmap changed)... 2024-02-10T06:39:32.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:32 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:39:32.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:32 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:32.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:32 smithi008 ceph-mon[24646]: Reconfiguring daemon crash.smithi008 on smithi008 2024-02-10T06:39:32.435 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:32 smithi029 ceph-mon[27077]: from='osd.0 [v2:172.21.15.8:6802/903496854,v1:172.21.15.8:6803/903496854]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T06:39:32.435 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:32 smithi029 ceph-mon[27077]: osdmap e8: 1 total, 0 up, 1 in 2024-02-10T06:39:32.435 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:32 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:39:32.435 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:32 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:39:32.435 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:32 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:32.435 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:32 smithi029 ceph-mon[27077]: Reconfiguring crash.smithi008 (monmap changed)... 2024-02-10T06:39:32.435 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:32 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:39:32.435 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:32 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:32.435 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:32 smithi029 ceph-mon[27077]: Reconfiguring daemon crash.smithi008 on smithi008 2024-02-10T06:39:32.967 INFO:teuthology.orchestra.run.smithi008.stdout:Created osd(s) 0 on host 'smithi008' 2024-02-10T06:39:33.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:33 smithi008 ceph-mon[24646]: purged_snaps scrub starts 2024-02-10T06:39:33.270 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:33 smithi008 ceph-mon[24646]: purged_snaps scrub ok 2024-02-10T06:39:33.270 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:33 smithi008 ceph-mon[24646]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:33.270 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:33 smithi008 ceph-mon[24646]: osd.0 [v2:172.21.15.8:6802/903496854,v1:172.21.15.8:6803/903496854] boot 2024-02-10T06:39:33.270 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:33 smithi008 ceph-mon[24646]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T06:39:33.270 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:33 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:39:33.270 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:33 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:33 smithi029 ceph-mon[27077]: purged_snaps scrub starts 2024-02-10T06:39:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:33 smithi029 ceph-mon[27077]: purged_snaps scrub ok 2024-02-10T06:39:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:33 smithi029 ceph-mon[27077]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:39:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:33 smithi029 ceph-mon[27077]: osd.0 [v2:172.21.15.8:6802/903496854,v1:172.21.15.8:6803/903496854] boot 2024-02-10T06:39:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:33 smithi029 ceph-mon[27077]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T06:39:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:33 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:39:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:33 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:34.233 DEBUG:teuthology.orchestra.run.smithi008:osd.0> sudo journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.0.service 2024-02-10T06:39:34.235 INFO:tasks.cephadm:Deploying osd.1 on smithi008 with /dev/vg_nvme/lv_3... 2024-02-10T06:39:34.236 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T06:39:35.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:35 smithi008 ceph-mon[24646]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:35.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:35 smithi008 ceph-mon[24646]: osdmap e10: 1 total, 1 up, 1 in 2024-02-10T06:39:35.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:35 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:35.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:35 smithi029 ceph-mon[27077]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:35.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:35 smithi029 ceph-mon[27077]: osdmap e10: 1 total, 1 up, 1 in 2024-02-10T06:39:35.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:35 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:36.087 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:36 smithi008 ceph-mon[24646]: Reconfiguring grafana.smithi008 (dependencies changed)... 2024-02-10T06:39:36.087 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:36 smithi008 ceph-mon[24646]: Reconfiguring daemon grafana.smithi008 on smithi008 2024-02-10T06:39:36.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:36 smithi029 ceph-mon[27077]: Reconfiguring grafana.smithi008 (dependencies changed)... 2024-02-10T06:39:36.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:36 smithi029 ceph-mon[27077]: Reconfiguring daemon grafana.smithi008 on smithi008 2024-02-10T06:39:37.265 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T06:39:37.265 INFO:teuthology.orchestra.run.smithi008.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T06:39:37.265 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10+0 records in 2024-02-10T06:39:37.265 INFO:teuthology.orchestra.run.smithi008.stderr:10+0 records out 2024-02-10T06:39:37.265 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00855128 s, 1.2 GB/s 2024-02-10T06:39:37.265 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping successful for: 2024-02-10T06:39:37.266 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:37 smithi008 ceph-mon[24646]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:37.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:37 smithi029 ceph-mon[27077]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:37.958 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi008:vg_nvme/lv_3 2024-02-10T06:39:39.258 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:39 smithi008 ceph-mon[24646]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:39.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:39 smithi029 ceph-mon[27077]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:41.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:41 smithi008 ceph-mon[24646]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:41.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:41 smithi029 ceph-mon[27077]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:42.126 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:42 smithi008 ceph-mon[24646]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:39:42.126 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:42 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:39:42.126 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:42 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:39:42.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:42 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:42.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:42 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:42.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:42 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:39:42.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:42 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:42.394 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:42 smithi029 ceph-mon[27077]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:39:42.394 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:42 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:39:42.394 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:42 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:39:42.394 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:42 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:42.394 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:42 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:42.394 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:42 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:39:42.394 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:42 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:43.117 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:43 smithi008 ceph-mon[24646]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:43.117 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:43 smithi008 ceph-mon[24646]: Reconfiguring crash.smithi029 (monmap changed)... 2024-02-10T06:39:43.117 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:43 smithi008 ceph-mon[24646]: Reconfiguring daemon crash.smithi029 on smithi029 2024-02-10T06:39:43.434 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:43 smithi029 ceph-mon[27077]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:43.435 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:43 smithi029 ceph-mon[27077]: Reconfiguring crash.smithi029 (monmap changed)... 2024-02-10T06:39:43.435 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:43 smithi029 ceph-mon[27077]: Reconfiguring daemon crash.smithi029 on smithi029 2024-02-10T06:39:44.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:44.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:44 smithi029 ceph-mon[27077]: Reconfiguring mgr.smithi029.fuvrtf (monmap changed)... 2024-02-10T06:39:44.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi029.fuvrtf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:39:44.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:39:44.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:44.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:44 smithi029 ceph-mon[27077]: Reconfiguring daemon mgr.smithi029.fuvrtf on smithi029 2024-02-10T06:39:44.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:44 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1558055322' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2869b5b-c167-4772-93fa-a2b817026314"}]: dispatch 2024-02-10T06:39:44.782 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:44 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1558055322' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2869b5b-c167-4772-93fa-a2b817026314"}]': finished 2024-02-10T06:39:44.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:44 smithi029 ceph-mon[27077]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T06:39:44.783 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:39:44.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:44.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:44 smithi008 ceph-mon[24646]: Reconfiguring mgr.smithi029.fuvrtf (monmap changed)... 2024-02-10T06:39:44.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi029.fuvrtf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:39:44.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:39:44.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:44.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:44 smithi008 ceph-mon[24646]: Reconfiguring daemon mgr.smithi029.fuvrtf on smithi029 2024-02-10T06:39:44.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:44 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1558055322' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2869b5b-c167-4772-93fa-a2b817026314"}]: dispatch 2024-02-10T06:39:44.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:44 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1558055322' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2869b5b-c167-4772-93fa-a2b817026314"}]': finished 2024-02-10T06:39:44.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:44 smithi008 ceph-mon[24646]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T06:39:44.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:39:45.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:45 smithi008 ceph-mon[24646]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:45.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:45 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/2066311432' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:39:45.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:45 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:45.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:45 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:39:45.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:45 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:39:45.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:45 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:45.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:45 smithi029 ceph-mon[27077]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:45.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:45 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/2066311432' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:39:45.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:45 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:45.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:45 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:39:45.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:45 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:39:45.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:45 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:46.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:46 smithi008 ceph-mon[24646]: Reconfiguring mon.smithi029 (monmap changed)... 2024-02-10T06:39:46.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:46 smithi008 ceph-mon[24646]: Reconfiguring daemon mon.smithi029 on smithi029 2024-02-10T06:39:46.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:46 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:46.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:46 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:39:46.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:46 smithi029 ceph-mon[27077]: Reconfiguring mon.smithi029 (monmap changed)... 2024-02-10T06:39:46.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:46 smithi029 ceph-mon[27077]: Reconfiguring daemon mon.smithi029 on smithi029 2024-02-10T06:39:46.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:46 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:46.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:46 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:39:47.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:47 smithi008 ceph-mon[24646]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:47.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:47 smithi008 ceph-mon[24646]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:39:47.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:47 smithi029 ceph-mon[27077]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:47.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:47 smithi029 ceph-mon[27077]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:39:49.611 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:49 smithi008 ceph-mon[24646]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:49.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:49 smithi029 ceph-mon[27077]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:51.580 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:51 smithi008 ceph-mon[24646]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:51.580 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:51 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T06:39:51.580 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:51 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:51.580 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:51 smithi008 ceph-mon[24646]: Deploying daemon osd.1 on smithi008 2024-02-10T06:39:51.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:51 smithi029 ceph-mon[27077]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:51.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:51 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T06:39:51.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:51 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:51.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:51 smithi029 ceph-mon[27077]: Deploying daemon osd.1 on smithi008 2024-02-10T06:39:53.756 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:53 smithi008 ceph-mon[24646]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:53.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:53 smithi029 ceph-mon[27077]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:55.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:54 smithi008 ceph-mon[24646]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:54 smithi029 ceph-mon[27077]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:56.856 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:56 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:56.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:56 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:57.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:57 smithi029 ceph-mon[27077]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:58.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:57 smithi008 ceph-mon[24646]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:58.311 INFO:teuthology.orchestra.run.smithi008.stdout:Created osd(s) 1 on host 'smithi008' 2024-02-10T06:39:59.084 DEBUG:teuthology.orchestra.run.smithi008:osd.1> sudo journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.1.service 2024-02-10T06:39:59.086 INFO:tasks.cephadm:Deploying osd.2 on smithi008 with /dev/vg_nvme/lv_2... 2024-02-10T06:39:59.086 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T06:39:59.613 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:59.613 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:59.613 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.8:9093"}]: dispatch 2024-02-10T06:39:59.613 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.8:9093"}]: dispatch 2024-02-10T06:39:59.613 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:59.613 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:39:59.613 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:39:59.614 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.8:3000"}]: dispatch 2024-02-10T06:39:59.614 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.8:3000"}]: dispatch 2024-02-10T06:39:59.614 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:59.614 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:39:59.614 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:39:59.614 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.8:9095"}]: dispatch 2024-02-10T06:39:59.614 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.8:9095"}]: dispatch 2024-02-10T06:39:59.614 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:59.614 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:39:59.614 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:59.614 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:39:59.614 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:39:59 smithi029 ceph-mon[27077]: from='osd.1 [v2:172.21.15.8:6810/3571451027,v1:172.21.15.8:6811/3571451027]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T06:39:59.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:39:59.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:59.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.8:9093"}]: dispatch 2024-02-10T06:39:59.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.8:9093"}]: dispatch 2024-02-10T06:39:59.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:59.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:39:59.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:39:59.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.8:3000"}]: dispatch 2024-02-10T06:39:59.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.8:3000"}]: dispatch 2024-02-10T06:39:59.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:59.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:39:59.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:39:59.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.8:9095"}]: dispatch 2024-02-10T06:39:59.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.8:9095"}]: dispatch 2024-02-10T06:39:59.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:39:59.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:39:59.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:39:59.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:39:59.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:39:59 smithi008 ceph-mon[24646]: from='osd.1 [v2:172.21.15.8:6810/3571451027,v1:172.21.15.8:6811/3571451027]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T06:40:00.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:00 smithi029 ceph-mon[27077]: from='osd.1 [v2:172.21.15.8:6810/3571451027,v1:172.21.15.8:6811/3571451027]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T06:40:00.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:00 smithi029 ceph-mon[27077]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T06:40:00.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:00 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:40:00.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:00 smithi029 ceph-mon[27077]: from='osd.1 [v2:172.21.15.8:6810/3571451027,v1:172.21.15.8:6811/3571451027]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T06:40:00.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:00 smithi029 ceph-mon[27077]: overall HEALTH_OK 2024-02-10T06:40:00.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:00 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:00.946 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:00 smithi008 ceph-mon[24646]: from='osd.1 [v2:172.21.15.8:6810/3571451027,v1:172.21.15.8:6811/3571451027]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T06:40:00.946 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:00 smithi008 ceph-mon[24646]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T06:40:00.946 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:00 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:40:00.946 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:00 smithi008 ceph-mon[24646]: from='osd.1 [v2:172.21.15.8:6810/3571451027,v1:172.21.15.8:6811/3571451027]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T06:40:00.946 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:00 smithi008 ceph-mon[24646]: overall HEALTH_OK 2024-02-10T06:40:00.946 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:00 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:00.946 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:40:00 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1[37910]: 2024-02-10T06:40:00.574+0000 7fd0285d4700 -1 osd.1 0 waiting for initial osdmap 2024-02-10T06:40:00.946 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:40:00 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1[37910]: 2024-02-10T06:40:00.577+0000 7fd02276a700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:40:01.682 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T06:40:01.682 INFO:teuthology.orchestra.run.smithi008.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T06:40:01.682 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10+0 records in 2024-02-10T06:40:01.682 INFO:teuthology.orchestra.run.smithi008.stderr:10+0 records out 2024-02-10T06:40:01.682 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0093157 s, 1.1 GB/s 2024-02-10T06:40:01.682 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping successful for: 2024-02-10T06:40:01.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:01 smithi008 ceph-mon[24646]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:40:01.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:01 smithi008 ceph-mon[24646]: from='osd.1 [v2:172.21.15.8:6810/3571451027,v1:172.21.15.8:6811/3571451027]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T06:40:01.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:01 smithi008 ceph-mon[24646]: osdmap e13: 2 total, 1 up, 2 in 2024-02-10T06:40:01.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:01 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:40:01.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:01 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:40:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:01 smithi029 ceph-mon[27077]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:40:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:01 smithi029 ceph-mon[27077]: from='osd.1 [v2:172.21.15.8:6810/3571451027,v1:172.21.15.8:6811/3571451027]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T06:40:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:01 smithi029 ceph-mon[27077]: osdmap e13: 2 total, 1 up, 2 in 2024-02-10T06:40:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:01 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:40:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:01 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:40:02.638 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi008:vg_nvme/lv_2 2024-02-10T06:40:02.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:02 smithi008 ceph-mon[24646]: purged_snaps scrub starts 2024-02-10T06:40:02.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:02 smithi008 ceph-mon[24646]: purged_snaps scrub ok 2024-02-10T06:40:02.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:02 smithi008 ceph-mon[24646]: osd.1 [v2:172.21.15.8:6810/3571451027,v1:172.21.15.8:6811/3571451027] boot 2024-02-10T06:40:02.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:02 smithi008 ceph-mon[24646]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T06:40:02.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:02 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:40:02.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:02 smithi029 ceph-mon[27077]: purged_snaps scrub starts 2024-02-10T06:40:02.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:02 smithi029 ceph-mon[27077]: purged_snaps scrub ok 2024-02-10T06:40:02.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:02 smithi029 ceph-mon[27077]: osd.1 [v2:172.21.15.8:6810/3571451027,v1:172.21.15.8:6811/3571451027] boot 2024-02-10T06:40:02.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:02 smithi029 ceph-mon[27077]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T06:40:02.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:02 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:40:03.764 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:03 smithi008 ceph-mon[24646]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:40:03.764 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:03 smithi008 ceph-mon[24646]: osdmap e15: 2 total, 2 up, 2 in 2024-02-10T06:40:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:03 smithi029 ceph-mon[27077]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:40:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:03 smithi029 ceph-mon[27077]: osdmap e15: 2 total, 2 up, 2 in 2024-02-10T06:40:05.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:04 smithi008 ceph-mon[24646]: pgmap v53: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:05.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:04 smithi029 ceph-mon[27077]: pgmap v53: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:06.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:05 smithi008 ceph-mon[24646]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:40:06.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:05 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:40:06.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:05 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:40:06.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:05 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:05 smithi029 ceph-mon[27077]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:40:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:05 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:40:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:05 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:40:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:05 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:07.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:06 smithi008 ceph-mon[24646]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:07.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:06 smithi029 ceph-mon[27077]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:08.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:08 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:08.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:08 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1221213386' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffae79d2-47bf-45d3-8d66-0cc121397b01"}]: dispatch 2024-02-10T06:40:08.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:08 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1221213386' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ffae79d2-47bf-45d3-8d66-0cc121397b01"}]': finished 2024-02-10T06:40:08.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:08 smithi008 ceph-mon[24646]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T06:40:08.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:08 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:40:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:08 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:08 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1221213386' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffae79d2-47bf-45d3-8d66-0cc121397b01"}]: dispatch 2024-02-10T06:40:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:08 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1221213386' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ffae79d2-47bf-45d3-8d66-0cc121397b01"}]': finished 2024-02-10T06:40:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:08 smithi029 ceph-mon[27077]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T06:40:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:08 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:40:09.623 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:09 smithi008 ceph-mon[24646]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:09.623 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:09 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/3098729015' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:40:09.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:09 smithi029 ceph-mon[27077]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:09.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:09 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/3098729015' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:40:11.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:11 smithi008 ceph-mon[24646]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:11.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:11 smithi029 ceph-mon[27077]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:13.647 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:13 smithi008 ceph-mon[24646]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:13.715 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:13 smithi029 ceph-mon[27077]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:15.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:14 smithi029 ceph-mon[27077]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:15.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:14 smithi029 ceph-mon[27077]: Detected new or changed devices on smithi008 2024-02-10T06:40:15.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:14 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:15.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:14 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:40:15.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:14 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:40:15.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:14 smithi029 ceph-mon[27077]: Adjusting osd_memory_target on smithi008 to 6434M 2024-02-10T06:40:15.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:14 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:15.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:14 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:15.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:14 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:40:15.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:14 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:15.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:14 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:40:15.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:14 smithi008 ceph-mon[24646]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:14 smithi008 ceph-mon[24646]: Detected new or changed devices on smithi008 2024-02-10T06:40:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:14 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:14 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:40:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:14 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:40:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:14 smithi008 ceph-mon[24646]: Adjusting osd_memory_target on smithi008 to 6434M 2024-02-10T06:40:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:14 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:14 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:14 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:40:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:14 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:14 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:40:16.748 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:16 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:16.748 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:16 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T06:40:16.748 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:16 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:16.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:16 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:16.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:16 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T06:40:16.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:16 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:17.678 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:17 smithi008 ceph-mon[24646]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:17.678 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:17 smithi008 ceph-mon[24646]: Deploying daemon osd.2 on smithi008 2024-02-10T06:40:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:17 smithi029 ceph-mon[27077]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:17 smithi029 ceph-mon[27077]: Deploying daemon osd.2 on smithi008 2024-02-10T06:40:19.949 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:19 smithi008 ceph-mon[24646]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:19.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:19 smithi029 ceph-mon[27077]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:21.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:21 smithi029 ceph-mon[27077]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:21.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:21 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:21.972 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:21 smithi008 ceph-mon[24646]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:21.972 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:21 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:23.945 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:23 smithi008 ceph-mon[24646]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:23.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:23 smithi029 ceph-mon[27077]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:24.994 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:24 smithi008 ceph-mon[24646]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:24.995 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:40:24.995 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:40:24.995 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:24 smithi029 ceph-mon[27077]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:40:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:40:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:27.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:27 smithi008 ceph-mon[24646]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:27.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:27 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:27.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:27 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:27.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:27 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:40:27.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:27 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:27.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:27 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:40:27.368 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:27 smithi029 ceph-mon[27077]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:27.368 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:27 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:27.368 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:27 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:27.368 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:27 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:40:27.368 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:27 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:27.368 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:27 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:40:27.947 INFO:teuthology.orchestra.run.smithi008.stdout:Created osd(s) 2 on host 'smithi008' 2024-02-10T06:40:28.246 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:28 smithi008 ceph-mon[24646]: from='osd.2 [v2:172.21.15.8:6818/2223271854,v1:172.21.15.8:6819/2223271854]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T06:40:28.246 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:28 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:28.246 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:28 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:28.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:28 smithi029 ceph-mon[27077]: from='osd.2 [v2:172.21.15.8:6818/2223271854,v1:172.21.15.8:6819/2223271854]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T06:40:28.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:28 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:28.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:28 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:28.547 DEBUG:teuthology.orchestra.run.smithi008:osd.2> sudo journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.2.service 2024-02-10T06:40:28.589 INFO:tasks.cephadm:Deploying osd.3 on smithi008 with /dev/vg_nvme/lv_1... 2024-02-10T06:40:28.589 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-10T06:40:29.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:29 smithi008 ceph-mon[24646]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:29.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:29 smithi008 ceph-mon[24646]: from='osd.2 [v2:172.21.15.8:6818/2223271854,v1:172.21.15.8:6819/2223271854]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T06:40:29.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:29 smithi008 ceph-mon[24646]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T06:40:29.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:29 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:40:29.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:29 smithi008 ceph-mon[24646]: from='osd.2 [v2:172.21.15.8:6818/2223271854,v1:172.21.15.8:6819/2223271854]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T06:40:29.284 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:40:29 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2[41772]: 2024-02-10T06:40:29.014+0000 7f4e43c0b700 -1 osd.2 0 waiting for initial osdmap 2024-02-10T06:40:29.284 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:40:29 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2[41772]: 2024-02-10T06:40:29.017+0000 7f4e405a6700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:40:29.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:29 smithi029 ceph-mon[27077]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:29.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:29 smithi029 ceph-mon[27077]: from='osd.2 [v2:172.21.15.8:6818/2223271854,v1:172.21.15.8:6819/2223271854]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T06:40:29.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:29 smithi029 ceph-mon[27077]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T06:40:29.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:29 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:40:29.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:29 smithi029 ceph-mon[27077]: from='osd.2 [v2:172.21.15.8:6818/2223271854,v1:172.21.15.8:6819/2223271854]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T06:40:30.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:30 smithi008 ceph-mon[24646]: from='osd.2 [v2:172.21.15.8:6818/2223271854,v1:172.21.15.8:6819/2223271854]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T06:40:30.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:30 smithi008 ceph-mon[24646]: osdmap e18: 3 total, 2 up, 3 in 2024-02-10T06:40:30.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:30 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:40:30.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:30 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:40:30.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:30 smithi029 ceph-mon[27077]: from='osd.2 [v2:172.21.15.8:6818/2223271854,v1:172.21.15.8:6819/2223271854]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T06:40:30.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:30 smithi029 ceph-mon[27077]: osdmap e18: 3 total, 2 up, 3 in 2024-02-10T06:40:30.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:30 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:40:30.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:30 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:40:31.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:31 smithi008 ceph-mon[24646]: purged_snaps scrub starts 2024-02-10T06:40:31.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:31 smithi008 ceph-mon[24646]: purged_snaps scrub ok 2024-02-10T06:40:31.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:31 smithi008 ceph-mon[24646]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:31.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:31 smithi008 ceph-mon[24646]: osd.2 [v2:172.21.15.8:6818/2223271854,v1:172.21.15.8:6819/2223271854] boot 2024-02-10T06:40:31.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:31 smithi008 ceph-mon[24646]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T06:40:31.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:31 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:40:31.444 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-10T06:40:31.444 INFO:teuthology.orchestra.run.smithi008.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-10T06:40:31.444 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10+0 records in 2024-02-10T06:40:31.444 INFO:teuthology.orchestra.run.smithi008.stderr:10+0 records out 2024-02-10T06:40:31.445 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00958289 s, 1.1 GB/s 2024-02-10T06:40:31.445 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping successful for: 2024-02-10T06:40:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:31 smithi029 ceph-mon[27077]: purged_snaps scrub starts 2024-02-10T06:40:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:31 smithi029 ceph-mon[27077]: purged_snaps scrub ok 2024-02-10T06:40:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:31 smithi029 ceph-mon[27077]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T06:40:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:31 smithi029 ceph-mon[27077]: osd.2 [v2:172.21.15.8:6818/2223271854,v1:172.21.15.8:6819/2223271854] boot 2024-02-10T06:40:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:31 smithi029 ceph-mon[27077]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T06:40:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:31 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:40:32.162 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi008:vg_nvme/lv_1 2024-02-10T06:40:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:32 smithi029 ceph-mon[27077]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T06:40:32.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:32 smithi008 ceph-mon[24646]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T06:40:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:33 smithi029 ceph-mon[27077]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:33 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T06:40:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:33 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T06:40:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:33 smithi029 ceph-mon[27077]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T06:40:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:33 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:40:33.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:33 smithi008 ceph-mon[24646]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:33.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:33 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T06:40:33.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:33 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T06:40:33.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:33 smithi008 ceph-mon[24646]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T06:40:33.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:33 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:40:34.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:34 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T06:40:34.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:34 smithi008 ceph-mon[24646]: osdmap e22: 3 total, 3 up, 3 in 2024-02-10T06:40:34.534 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:40:34 smithi008 sudo[43167]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T06:40:34.534 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:40:34 smithi008 sudo[43167]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:40:34.534 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:40:34 smithi008 sudo[43167]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:40:34.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:34 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T06:40:34.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:34 smithi029 ceph-mon[27077]: osdmap e22: 3 total, 3 up, 3 in 2024-02-10T06:40:34.945 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:40:34 smithi008 sudo[43195]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T06:40:34.945 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:40:34 smithi008 sudo[43195]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:40:34.945 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:40:34 smithi008 sudo[43195]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:40:35.212 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:40:34 smithi008 sudo[43275]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T06:40:35.212 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:40:34 smithi008 sudo[43275]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:40:35.213 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:40:35 smithi008 sudo[43275]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:40:35.533 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:40:35 smithi008 sudo[43305]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T06:40:35.533 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:40:35 smithi008 sudo[43305]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:40:35.533 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:40:35 smithi008 sudo[43305]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:40:35.825 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:40:35 smithi008 sudo[43372]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T06:40:35.826 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:40:35 smithi008 sudo[43372]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:40:35.826 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:40:35 smithi008 sudo[43372]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:40:35.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:35 smithi008 ceph-mon[24646]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:35.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:35 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:35.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:35 smithi008 ceph-mon[24646]: osdmap e23: 3 total, 3 up, 3 in 2024-02-10T06:40:35.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:35 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:40:35.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:35 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:40:35.827 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:35 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:35.827 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:35 smithi008 ceph-mon[24646]: mgrmap e19: smithi008.pxifbt(active, since 2m), standbys: smithi029.fuvrtf 2024-02-10T06:40:35.827 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:35 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:36.092 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:40:35 smithi008 sudo[43446]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T06:40:36.092 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:40:35 smithi008 sudo[43446]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:40:36.092 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:40:35 smithi008 sudo[43446]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:40:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:35 smithi029 ceph-mon[27077]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:35 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:35 smithi029 ceph-mon[27077]: osdmap e23: 3 total, 3 up, 3 in 2024-02-10T06:40:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:35 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:40:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:35 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:40:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:35 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:35 smithi029 ceph-mon[27077]: mgrmap e19: smithi008.pxifbt(active, since 2m), standbys: smithi029.fuvrtf 2024-02-10T06:40:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:35 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:36.358 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:36 smithi008 sudo[43475]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T06:40:36.358 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:36 smithi008 sudo[43475]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:40:36.358 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:36 smithi008 sudo[43475]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:40:36.611 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:36 smithi008 sudo[43481]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T06:40:36.611 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:36 smithi008 sudo[43481]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:40:36.611 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:36 smithi008 sudo[43481]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:40:36.876 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:36 smithi029 sudo[29375]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T06:40:36.877 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:36 smithi029 sudo[29375]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:40:36.877 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:36 smithi029 sudo[29375]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:40:37.184 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:36 smithi008 ceph-mon[24646]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:40:37.184 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:36 smithi008 ceph-mon[24646]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T06:40:37.184 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:36 smithi008 ceph-mon[24646]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T06:40:37.185 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:36 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:40:37.185 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:36 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:40:37.185 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:36 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:40:37.185 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:36 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:40:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:36 smithi029 sudo[29378]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T06:40:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:36 smithi029 sudo[29378]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:40:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:36 smithi029 sudo[29378]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:40:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:36 smithi029 ceph-mon[27077]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:40:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:36 smithi029 ceph-mon[27077]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T06:40:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:36 smithi029 ceph-mon[27077]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T06:40:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:36 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:40:37.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:36 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:40:37.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:36 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:40:37.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:36 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:40:38.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:37 smithi008 ceph-mon[24646]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:38.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:37 smithi008 ceph-mon[24646]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T06:40:38.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:37 smithi008 ceph-mon[24646]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T06:40:38.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:37 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:37 smithi029 ceph-mon[27077]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:37 smithi029 ceph-mon[27077]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T06:40:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:37 smithi029 ceph-mon[27077]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T06:40:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:37 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:38 smithi029 ceph-mon[27077]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:38 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/3150735874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a73469d1-315f-41d5-81d1-b9c69df34e8a"}]: dispatch 2024-02-10T06:40:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:38 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/3150735874' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a73469d1-315f-41d5-81d1-b9c69df34e8a"}]': finished 2024-02-10T06:40:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:38 smithi029 ceph-mon[27077]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T06:40:39.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:38 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:40:39.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:38 smithi008 ceph-mon[24646]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:39.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:38 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/3150735874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a73469d1-315f-41d5-81d1-b9c69df34e8a"}]: dispatch 2024-02-10T06:40:39.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:38 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/3150735874' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a73469d1-315f-41d5-81d1-b9c69df34e8a"}]': finished 2024-02-10T06:40:39.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:38 smithi008 ceph-mon[24646]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T06:40:39.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:38 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:40:40.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:39 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/576947495' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:40:40.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:39 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/576947495' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:40:41.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:40 smithi029 ceph-mon[27077]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:41.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:40 smithi008 ceph-mon[24646]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:43.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:42 smithi029 ceph-mon[27077]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:43.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:42 smithi008 ceph-mon[24646]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:45.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:44 smithi008 ceph-mon[24646]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:45.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:44 smithi008 ceph-mon[24646]: Detected new or changed devices on smithi008 2024-02-10T06:40:45.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:45.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:40:45.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:40:45.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:40:45.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:44 smithi008 ceph-mon[24646]: Adjusting osd_memory_target on smithi008 to 4289M 2024-02-10T06:40:45.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:45.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:45.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:40:45.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:45.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:40:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:44 smithi029 ceph-mon[27077]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:44 smithi029 ceph-mon[27077]: Detected new or changed devices on smithi008 2024-02-10T06:40:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:40:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:40:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:40:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:44 smithi029 ceph-mon[27077]: Adjusting osd_memory_target on smithi008 to 4289M 2024-02-10T06:40:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:45.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:45.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:40:45.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:45.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:40:46.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:45 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T06:40:46.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:45 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:46.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:45 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T06:40:46.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:45 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:47.277 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:46 smithi008 ceph-mon[24646]: Deploying daemon osd.3 on smithi008 2024-02-10T06:40:47.277 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:46 smithi008 ceph-mon[24646]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:47.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:46 smithi029 ceph-mon[27077]: Deploying daemon osd.3 on smithi008 2024-02-10T06:40:47.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:46 smithi029 ceph-mon[27077]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:49.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:48 smithi008 ceph-mon[24646]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:49.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:48 smithi029 ceph-mon[27077]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:51.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:50 smithi008 ceph-mon[24646]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:51.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:50 smithi029 ceph-mon[27077]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:53.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:53 smithi008 ceph-mon[24646]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:53.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:53 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:53 smithi029 ceph-mon[27077]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:53 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:55.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:54 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:55.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:54 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:55.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:54 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:40:55.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:54 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:55.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:54 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:40:55.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:54 smithi008 ceph-mon[24646]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:54 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:54 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:54 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:40:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:54 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:40:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:54 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:40:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:54 smithi029 ceph-mon[27077]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:55.438 INFO:teuthology.orchestra.run.smithi008.stdout:Created osd(s) 3 on host 'smithi008' 2024-02-10T06:40:56.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:55 smithi008 ceph-mon[24646]: from='osd.3 [v2:172.21.15.8:6826/1037657938,v1:172.21.15.8:6827/1037657938]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:40:56.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:55 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:55 smithi029 ceph-mon[27077]: from='osd.3 [v2:172.21.15.8:6826/1037657938,v1:172.21.15.8:6827/1037657938]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:40:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:55 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:40:56.373 DEBUG:teuthology.orchestra.run.smithi008:osd.3> sudo journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.3.service 2024-02-10T06:40:56.375 INFO:tasks.cephadm:Deploying osd.4 on smithi029 with /dev/vg_nvme/lv_4... 2024-02-10T06:40:56.375 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T06:40:56.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:56 smithi029 ceph-mon[27077]: from='osd.3 [v2:172.21.15.8:6826/1037657938,v1:172.21.15.8:6827/1037657938]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T06:40:56.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:56 smithi029 ceph-mon[27077]: osdmap e25: 4 total, 3 up, 4 in 2024-02-10T06:40:56.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:56 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:40:56.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:56 smithi029 ceph-mon[27077]: from='osd.3 [v2:172.21.15.8:6826/1037657938,v1:172.21.15.8:6827/1037657938]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T06:40:56.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:56 smithi029 ceph-mon[27077]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:57.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:56 smithi008 ceph-mon[24646]: from='osd.3 [v2:172.21.15.8:6826/1037657938,v1:172.21.15.8:6827/1037657938]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T06:40:57.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:56 smithi008 ceph-mon[24646]: osdmap e25: 4 total, 3 up, 4 in 2024-02-10T06:40:57.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:56 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:40:57.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:56 smithi008 ceph-mon[24646]: from='osd.3 [v2:172.21.15.8:6826/1037657938,v1:172.21.15.8:6827/1037657938]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T06:40:57.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:56 smithi008 ceph-mon[24646]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:57.033 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:40:56 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3[45519]: 2024-02-10T06:40:56.848+0000 7f01f6e17700 -1 osd.3 0 waiting for initial osdmap 2024-02-10T06:40:57.034 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:40:56 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3[45519]: 2024-02-10T06:40:56.852+0000 7f01f2fb1700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:40:58.003 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:57 smithi029 ceph-mon[27077]: from='osd.3 [v2:172.21.15.8:6826/1037657938,v1:172.21.15.8:6827/1037657938]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T06:40:58.003 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:57 smithi029 ceph-mon[27077]: osdmap e26: 4 total, 3 up, 4 in 2024-02-10T06:40:58.003 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:57 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:40:58.003 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:57 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:40:58.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:57 smithi008 ceph-mon[24646]: from='osd.3 [v2:172.21.15.8:6826/1037657938,v1:172.21.15.8:6827/1037657938]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T06:40:58.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:57 smithi008 ceph-mon[24646]: osdmap e26: 4 total, 3 up, 4 in 2024-02-10T06:40:58.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:57 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:40:58.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:57 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:40:58.485 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T06:40:58.485 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T06:40:58.485 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-02-10T06:40:58.486 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-02-10T06:40:58.486 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122202 s, 858 MB/s 2024-02-10T06:40:58.486 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-02-10T06:40:59.076 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:58 smithi029 ceph-mon[27077]: purged_snaps scrub starts 2024-02-10T06:40:59.077 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:58 smithi029 ceph-mon[27077]: purged_snaps scrub ok 2024-02-10T06:40:59.077 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:58 smithi029 ceph-mon[27077]: osd.3 [v2:172.21.15.8:6826/1037657938,v1:172.21.15.8:6827/1037657938] boot 2024-02-10T06:40:59.077 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:58 smithi029 ceph-mon[27077]: osdmap e27: 4 total, 4 up, 4 in 2024-02-10T06:40:59.077 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:58 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:40:59.077 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:58 smithi029 ceph-mon[27077]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:59.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_4 2024-02-10T06:40:59.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:58 smithi008 ceph-mon[24646]: purged_snaps scrub starts 2024-02-10T06:40:59.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:58 smithi008 ceph-mon[24646]: purged_snaps scrub ok 2024-02-10T06:40:59.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:58 smithi008 ceph-mon[24646]: osd.3 [v2:172.21.15.8:6826/1037657938,v1:172.21.15.8:6827/1037657938] boot 2024-02-10T06:40:59.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:58 smithi008 ceph-mon[24646]: osdmap e27: 4 total, 4 up, 4 in 2024-02-10T06:40:59.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:58 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:40:59.172 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:58 smithi008 ceph-mon[24646]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T06:40:59.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:59 smithi029 ceph-mon[27077]: osdmap e28: 4 total, 4 up, 4 in 2024-02-10T06:40:59.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:40:59 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:00.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:59 smithi008 ceph-mon[24646]: osdmap e28: 4 total, 4 up, 4 in 2024-02-10T06:41:00.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:40:59 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:00.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:00 smithi029 ceph-mon[27077]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:01.170 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:00 smithi008 ceph-mon[24646]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:01 smithi029 ceph-mon[27077]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:41:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:01 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:41:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:01 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:41:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:01 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:02.259 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:01 smithi008 ceph-mon[24646]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:41:02.259 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:01 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:41:02.259 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:01 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:41:02.260 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:01 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:03.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:03.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:03.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:41:03.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:41:03.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:41:03.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:41:03.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: Adjusting osd_memory_target on smithi008 to 3217M 2024-02-10T06:41:03.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:03.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:03.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:41:03.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:03.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:41:03.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/3260849023' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1568e47a-0969-40b2-bf29-8a1ccc4d5752"}]: dispatch 2024-02-10T06:41:03.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1568e47a-0969-40b2-bf29-8a1ccc4d5752"}]: dispatch 2024-02-10T06:41:03.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1568e47a-0969-40b2-bf29-8a1ccc4d5752"}]': finished 2024-02-10T06:41:03.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T06:41:03.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:41:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:41:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:41:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:41:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:41:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: Adjusting osd_memory_target on smithi008 to 3217M 2024-02-10T06:41:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:03.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:03.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:41:03.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:03.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:41:03.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: from='client.? 172.21.15.29:0/3260849023' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1568e47a-0969-40b2-bf29-8a1ccc4d5752"}]: dispatch 2024-02-10T06:41:03.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1568e47a-0969-40b2-bf29-8a1ccc4d5752"}]: dispatch 2024-02-10T06:41:03.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1568e47a-0969-40b2-bf29-8a1ccc4d5752"}]': finished 2024-02-10T06:41:03.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T06:41:03.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:41:05.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:04 smithi008 ceph-mon[24646]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:05.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:04 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/1617677277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:41:05.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:04 smithi029 ceph-mon[27077]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:05.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:04 smithi029 ceph-mon[27077]: from='client.? 172.21.15.29:0/1617677277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:41:07.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:06 smithi029 ceph-mon[27077]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:07.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:06 smithi008 ceph-mon[24646]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:07 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:07 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:08.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:07 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:08.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:07 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:09.109 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:08 smithi029 ceph-mon[27077]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:09.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:08 smithi008 ceph-mon[24646]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:10.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:09 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T06:41:10.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:09 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:10.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:09 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T06:41:10.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:09 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:11.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:10 smithi029 ceph-mon[27077]: Deploying daemon osd.4 on smithi029 2024-02-10T06:41:11.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:10 smithi029 ceph-mon[27077]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:11.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:10 smithi008 ceph-mon[24646]: Deploying daemon osd.4 on smithi029 2024-02-10T06:41:11.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:10 smithi008 ceph-mon[24646]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:13.088 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:12 smithi029 ceph-mon[27077]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:13.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:12 smithi008 ceph-mon[24646]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:14.839 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:14 smithi029 ceph-mon[27077]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:15.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:14 smithi008 ceph-mon[24646]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:15 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:15 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:41:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:15 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:15 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:41:16.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:15 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:16.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:15 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:41:16.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:15 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:16.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:15 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:41:17.180 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:16 smithi029 ceph-mon[27077]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:17.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:16 smithi008 ceph-mon[24646]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:18.400 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 4 on host 'smithi029' 2024-02-10T06:41:18.983 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:18 smithi029 ceph-mon[27077]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:18.983 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:18 smithi029 ceph-mon[27077]: from='osd.4 [v2:172.21.15.29:6800/3369615219,v1:172.21.15.29:6801/3369615219]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:41:18.984 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:18 smithi029 ceph-mon[27077]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:41:18.984 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:18 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:19.011 DEBUG:teuthology.orchestra.run.smithi029:osd.4> sudo journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.4.service 2024-02-10T06:41:19.013 INFO:tasks.cephadm:Deploying osd.5 on smithi029 with /dev/vg_nvme/lv_3... 2024-02-10T06:41:19.014 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T06:41:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:18 smithi008 ceph-mon[24646]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:18 smithi008 ceph-mon[24646]: from='osd.4 [v2:172.21.15.29:6800/3369615219,v1:172.21.15.29:6801/3369615219]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:41:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:18 smithi008 ceph-mon[24646]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:41:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:18 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:20.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:19 smithi029 ceph-mon[27077]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T06:41:20.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:19 smithi029 ceph-mon[27077]: osdmap e30: 5 total, 4 up, 5 in 2024-02-10T06:41:20.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:19 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:41:20.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:19 smithi029 ceph-mon[27077]: from='osd.4 [v2:172.21.15.29:6800/3369615219,v1:172.21.15.29:6801/3369615219]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:41:20.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:19 smithi029 ceph-mon[27077]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:41:20.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:19 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:20.208 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:41:19 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[31321]: 2024-02-10T06:41:19.945+0000 7fd63ecc3700 -1 osd.4 0 waiting for initial osdmap 2024-02-10T06:41:20.209 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:41:19 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[31321]: 2024-02-10T06:41:19.950+0000 7fd638658700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:41:20.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:19 smithi008 ceph-mon[24646]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T06:41:20.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:19 smithi008 ceph-mon[24646]: osdmap e30: 5 total, 4 up, 5 in 2024-02-10T06:41:20.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:19 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:41:20.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:19 smithi008 ceph-mon[24646]: from='osd.4 [v2:172.21.15.29:6800/3369615219,v1:172.21.15.29:6801/3369615219]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:41:20.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:19 smithi008 ceph-mon[24646]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:41:20.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:19 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:21.019 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:20 smithi029 ceph-mon[27077]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:21.019 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:20 smithi029 ceph-mon[27077]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-02-10T06:41:21.019 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:20 smithi029 ceph-mon[27077]: osdmap e31: 5 total, 4 up, 5 in 2024-02-10T06:41:21.019 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:20 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:41:21.019 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:20 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:41:21.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:20 smithi008 ceph-mon[24646]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:21.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:20 smithi008 ceph-mon[24646]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-02-10T06:41:21.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:20 smithi008 ceph-mon[24646]: osdmap e31: 5 total, 4 up, 5 in 2024-02-10T06:41:21.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:20 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:41:21.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:20 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:41:21.657 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T06:41:21.657 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T06:41:21.657 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-02-10T06:41:21.657 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-02-10T06:41:21.657 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0100535 s, 1.0 GB/s 2024-02-10T06:41:21.657 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-02-10T06:41:22.133 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:21 smithi029 ceph-mon[27077]: purged_snaps scrub starts 2024-02-10T06:41:22.133 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:21 smithi029 ceph-mon[27077]: purged_snaps scrub ok 2024-02-10T06:41:22.133 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:21 smithi029 ceph-mon[27077]: osd.4 [v2:172.21.15.29:6800/3369615219,v1:172.21.15.29:6801/3369615219] boot 2024-02-10T06:41:22.134 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:21 smithi029 ceph-mon[27077]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T06:41:22.134 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:21 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:41:22.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:21 smithi008 ceph-mon[24646]: purged_snaps scrub starts 2024-02-10T06:41:22.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:21 smithi008 ceph-mon[24646]: purged_snaps scrub ok 2024-02-10T06:41:22.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:21 smithi008 ceph-mon[24646]: osd.4 [v2:172.21.15.29:6800/3369615219,v1:172.21.15.29:6801/3369615219] boot 2024-02-10T06:41:22.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:21 smithi008 ceph-mon[24646]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T06:41:22.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:21 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:41:23.046 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:22 smithi029 ceph-mon[27077]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:23.046 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:22 smithi029 ceph-mon[27077]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T06:41:23.046 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:22 smithi029 ceph-mon[27077]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T06:41:23.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_3 2024-02-10T06:41:23.205 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:22 smithi008 ceph-mon[24646]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T06:41:23.206 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:22 smithi008 ceph-mon[24646]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T06:41:23.206 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:22 smithi008 ceph-mon[24646]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T06:41:24.890 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:24 smithi029 ceph-mon[27077]: osdmap e35: 5 total, 5 up, 5 in 2024-02-10T06:41:24.890 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:24 smithi029 ceph-mon[27077]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T06:41:24.890 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:41:24.890 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:41:25.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:24 smithi008 ceph-mon[24646]: osdmap e35: 5 total, 5 up, 5 in 2024-02-10T06:41:25.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:24 smithi008 ceph-mon[24646]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T06:41:25.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:41:25.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:41:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:25 smithi029 ceph-mon[27077]: osdmap e36: 5 total, 5 up, 5 in 2024-02-10T06:41:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:25 smithi029 ceph-mon[27077]: Detected new or changed devices on smithi029 2024-02-10T06:41:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:25 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:25 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:41:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:25 smithi029 ceph-mon[27077]: Adjusting osd_memory_target on smithi029 to 15941M 2024-02-10T06:41:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:25 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:25 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:25 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:41:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:25 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:26.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:25 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:41:26.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:25 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:26.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:25 smithi008 ceph-mon[24646]: osdmap e36: 5 total, 5 up, 5 in 2024-02-10T06:41:26.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:25 smithi008 ceph-mon[24646]: Detected new or changed devices on smithi029 2024-02-10T06:41:26.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:25 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:26.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:25 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:41:26.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:25 smithi008 ceph-mon[24646]: Adjusting osd_memory_target on smithi029 to 15941M 2024-02-10T06:41:26.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:25 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:26.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:25 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:26.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:25 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:41:26.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:25 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:26.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:25 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:41:26.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:25 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:26.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:26 smithi029 ceph-mon[27077]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T06:41:26.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:26 smithi029 ceph-mon[27077]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:41:26.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:26 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:41:26.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:26 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:41:26.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:26 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:27.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:26 smithi008 ceph-mon[24646]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T06:41:27.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:26 smithi008 ceph-mon[24646]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:41:27.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:26 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:41:27.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:26 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:41:27.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:26 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:29.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:28 smithi029 ceph-mon[27077]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-02-10T06:41:29.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:28 smithi029 ceph-mon[27077]: from='client.? 172.21.15.29:0/1521719735' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91c1b854-0a37-4643-9a32-0307c6c1bfd9"}]: dispatch 2024-02-10T06:41:29.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:28 smithi029 ceph-mon[27077]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91c1b854-0a37-4643-9a32-0307c6c1bfd9"}]: dispatch 2024-02-10T06:41:29.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:28 smithi029 ceph-mon[27077]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91c1b854-0a37-4643-9a32-0307c6c1bfd9"}]': finished 2024-02-10T06:41:29.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:28 smithi029 ceph-mon[27077]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T06:41:29.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:28 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:41:29.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:28 smithi008 ceph-mon[24646]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-02-10T06:41:29.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:28 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/1521719735' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91c1b854-0a37-4643-9a32-0307c6c1bfd9"}]: dispatch 2024-02-10T06:41:29.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:28 smithi008 ceph-mon[24646]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91c1b854-0a37-4643-9a32-0307c6c1bfd9"}]: dispatch 2024-02-10T06:41:29.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:28 smithi008 ceph-mon[24646]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91c1b854-0a37-4643-9a32-0307c6c1bfd9"}]': finished 2024-02-10T06:41:29.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:28 smithi008 ceph-mon[24646]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T06:41:29.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:28 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:41:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:29 smithi029 ceph-mon[27077]: from='client.? 172.21.15.29:0/2448151944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:41:30.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:29 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/2448151944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:41:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:30 smithi029 ceph-mon[27077]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-10T06:41:31.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:30 smithi008 ceph-mon[24646]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-10T06:41:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:32 smithi029 ceph-mon[27077]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T06:41:33.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:32 smithi008 ceph-mon[24646]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T06:41:34.924 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:34 smithi029 ceph-mon[27077]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-02-10T06:41:35.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:34 smithi008 ceph-mon[24646]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-02-10T06:41:36.068 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:35 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T06:41:36.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:35 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:36.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:35 smithi029 ceph-mon[27077]: Deploying daemon osd.5 on smithi029 2024-02-10T06:41:36.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:35 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T06:41:36.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:35 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:36.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:35 smithi008 ceph-mon[24646]: Deploying daemon osd.5 on smithi029 2024-02-10T06:41:36.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:36 smithi029 ceph-mon[27077]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T06:41:37.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:36 smithi008 ceph-mon[24646]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T06:41:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:38 smithi029 ceph-mon[27077]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T06:41:39.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:38 smithi008 ceph-mon[24646]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T06:41:41.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:41 smithi029 ceph-mon[27077]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T06:41:41.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:41 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:41.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:41 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:41:41.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:41 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:41.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:41 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:41:41.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:41 smithi008 ceph-mon[24646]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T06:41:41.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:41 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:41.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:41 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:41:41.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:41 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:41.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:41 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:41:43.108 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 5 on host 'smithi029' 2024-02-10T06:41:43.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:43 smithi029 ceph-mon[27077]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T06:41:43.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:43 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:43.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:43 smithi008 ceph-mon[24646]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T06:41:43.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:43 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:43.696 DEBUG:teuthology.orchestra.run.smithi029:osd.5> sudo journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.5.service 2024-02-10T06:41:43.699 INFO:tasks.cephadm:Deploying osd.6 on smithi029 with /dev/vg_nvme/lv_2... 2024-02-10T06:41:43.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T06:41:44.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:44 smithi029 ceph-mon[27077]: from='osd.5 [v2:172.21.15.29:6808/4103839609,v1:172.21.15.29:6809/4103839609]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:41:44.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:44 smithi029 ceph-mon[27077]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:41:44.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:44 smithi008 ceph-mon[24646]: from='osd.5 [v2:172.21.15.29:6808/4103839609,v1:172.21.15.29:6809/4103839609]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:41:44.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:44 smithi008 ceph-mon[24646]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:41:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:45 smithi029 ceph-mon[27077]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T06:41:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:45 smithi029 ceph-mon[27077]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T06:41:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:45 smithi029 ceph-mon[27077]: osdmap e38: 6 total, 5 up, 6 in 2024-02-10T06:41:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:45 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:41:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:45 smithi029 ceph-mon[27077]: from='osd.5 [v2:172.21.15.29:6808/4103839609,v1:172.21.15.29:6809/4103839609]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:41:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:45 smithi029 ceph-mon[27077]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:41:45.459 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:41:45 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[34293]: 2024-02-10T06:41:45.205+0000 7f8b9f5b0700 -1 osd.5 0 waiting for initial osdmap 2024-02-10T06:41:45.459 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:41:45 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[34293]: 2024-02-10T06:41:45.211+0000 7f8b9bf4b700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:41:45.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:45 smithi008 ceph-mon[24646]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T06:41:45.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:45 smithi008 ceph-mon[24646]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T06:41:45.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:45 smithi008 ceph-mon[24646]: osdmap e38: 6 total, 5 up, 6 in 2024-02-10T06:41:45.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:45 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:41:45.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:45 smithi008 ceph-mon[24646]: from='osd.5 [v2:172.21.15.29:6808/4103839609,v1:172.21.15.29:6809/4103839609]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:41:45.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:45 smithi008 ceph-mon[24646]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:41:46.376 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:46 smithi029 ceph-mon[27077]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-02-10T06:41:46.376 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:46 smithi029 ceph-mon[27077]: osdmap e39: 6 total, 5 up, 6 in 2024-02-10T06:41:46.376 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:46 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:41:46.376 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:46 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:41:46.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:46 smithi008 ceph-mon[24646]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-02-10T06:41:46.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:46 smithi008 ceph-mon[24646]: osdmap e39: 6 total, 5 up, 6 in 2024-02-10T06:41:46.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:46 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:41:46.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:46 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:41:46.656 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T06:41:46.656 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T06:41:46.656 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-02-10T06:41:46.657 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-02-10T06:41:46.657 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117422 s, 893 MB/s 2024-02-10T06:41:46.657 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-02-10T06:41:47.258 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_2 2024-02-10T06:41:47.429 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:47 smithi029 ceph-mon[27077]: purged_snaps scrub starts 2024-02-10T06:41:47.429 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:47 smithi029 ceph-mon[27077]: purged_snaps scrub ok 2024-02-10T06:41:47.430 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:47 smithi029 ceph-mon[27077]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T06:41:47.430 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:47 smithi029 ceph-mon[27077]: osd.5 [v2:172.21.15.29:6808/4103839609,v1:172.21.15.29:6809/4103839609] boot 2024-02-10T06:41:47.430 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:47 smithi029 ceph-mon[27077]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T06:41:47.430 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:47 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:41:47.430 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:47 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:47.430 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:47 smithi029 ceph-mon[27077]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T06:41:47.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:47 smithi008 ceph-mon[24646]: purged_snaps scrub starts 2024-02-10T06:41:47.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:47 smithi008 ceph-mon[24646]: purged_snaps scrub ok 2024-02-10T06:41:47.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:47 smithi008 ceph-mon[24646]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T06:41:47.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:47 smithi008 ceph-mon[24646]: osd.5 [v2:172.21.15.29:6808/4103839609,v1:172.21.15.29:6809/4103839609] boot 2024-02-10T06:41:47.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:47 smithi008 ceph-mon[24646]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T06:41:47.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:47 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:41:47.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:47 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:47.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:47 smithi008 ceph-mon[24646]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T06:41:49.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:49 smithi008 ceph-mon[24646]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T06:41:49.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:49 smithi008 ceph-mon[24646]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T06:41:49.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:49 smithi008 ceph-mon[24646]: osdmap e43: 6 total, 6 up, 6 in 2024-02-10T06:41:49.671 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:49 smithi029 ceph-mon[27077]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T06:41:49.671 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:49 smithi029 ceph-mon[27077]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T06:41:49.671 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:49 smithi029 ceph-mon[27077]: osdmap e43: 6 total, 6 up, 6 in 2024-02-10T06:41:51.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:50 smithi029 ceph-mon[27077]: osdmap e44: 6 total, 6 up, 6 in 2024-02-10T06:41:51.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:50 smithi029 ceph-mon[27077]: pgmap v135: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T06:41:51.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:50 smithi008 ceph-mon[24646]: osdmap e44: 6 total, 6 up, 6 in 2024-02-10T06:41:51.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:50 smithi008 ceph-mon[24646]: pgmap v135: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T06:41:51.916 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:51 smithi029 ceph-mon[27077]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T06:41:51.916 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:51 smithi029 ceph-mon[27077]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:41:51.917 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:51 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:41:51.917 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:51 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:41:51.917 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:51 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:51.917 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:51 smithi029 ceph-mon[27077]: Detected new or changed devices on smithi029 2024-02-10T06:41:51.917 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:51 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:51.917 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:51 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:41:51.917 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:51 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:41:51.917 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:51 smithi029 ceph-mon[27077]: Adjusting osd_memory_target on smithi029 to 7970M 2024-02-10T06:41:51.917 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:51 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:51.917 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:51 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:51.917 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:51 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:41:51.917 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:51 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:51.917 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:51 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:41:51.917 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:51 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:52.185 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:51 smithi008 ceph-mon[24646]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T06:41:52.185 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:51 smithi008 ceph-mon[24646]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:41:52.185 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:51 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:41:52.185 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:51 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:41:52.185 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:51 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:52.185 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:51 smithi008 ceph-mon[24646]: Detected new or changed devices on smithi029 2024-02-10T06:41:52.185 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:51 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:52.185 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:51 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:41:52.186 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:51 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:41:52.186 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:51 smithi008 ceph-mon[24646]: Adjusting osd_memory_target on smithi029 to 7970M 2024-02-10T06:41:52.186 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:51 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:52.186 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:51 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:52.186 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:51 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:41:52.186 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:51 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:41:52.186 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:51 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:41:52.186 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:51 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:41:52.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:52 smithi029 ceph-mon[27077]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T06:41:53.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:52 smithi008 ceph-mon[24646]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T06:41:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:53 smithi029 ceph-mon[27077]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T06:41:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:53 smithi029 ceph-mon[27077]: Cluster is now healthy 2024-02-10T06:41:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:53 smithi029 ceph-mon[27077]: from='client.? 172.21.15.29:0/395028445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d24e3ca9-b39f-4e81-855e-b3305151ad5e"}]: dispatch 2024-02-10T06:41:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:53 smithi029 ceph-mon[27077]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d24e3ca9-b39f-4e81-855e-b3305151ad5e"}]: dispatch 2024-02-10T06:41:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:53 smithi029 ceph-mon[27077]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d24e3ca9-b39f-4e81-855e-b3305151ad5e"}]': finished 2024-02-10T06:41:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:53 smithi029 ceph-mon[27077]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T06:41:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:53 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:41:54.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:53 smithi008 ceph-mon[24646]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T06:41:54.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:53 smithi008 ceph-mon[24646]: Cluster is now healthy 2024-02-10T06:41:54.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:53 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/395028445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d24e3ca9-b39f-4e81-855e-b3305151ad5e"}]: dispatch 2024-02-10T06:41:54.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:53 smithi008 ceph-mon[24646]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d24e3ca9-b39f-4e81-855e-b3305151ad5e"}]: dispatch 2024-02-10T06:41:54.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:53 smithi008 ceph-mon[24646]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d24e3ca9-b39f-4e81-855e-b3305151ad5e"}]': finished 2024-02-10T06:41:54.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:53 smithi008 ceph-mon[24646]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T06:41:54.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:53 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:41:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:54 smithi029 ceph-mon[27077]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-02-10T06:41:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:54 smithi029 ceph-mon[27077]: from='client.? 172.21.15.29:0/3443564948' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:41:55.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:54 smithi008 ceph-mon[24646]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-02-10T06:41:55.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:54 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/3443564948' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:41:57.160 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:56 smithi029 ceph-mon[27077]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-02-10T06:41:57.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:56 smithi008 ceph-mon[24646]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-02-10T06:41:59.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:41:58 smithi029 ceph-mon[27077]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-02-10T06:41:59.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:41:58 smithi008 ceph-mon[24646]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-02-10T06:42:01.142 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:00 smithi029 ceph-mon[27077]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T06:42:01.142 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:00 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T06:42:01.142 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:00 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:42:01.143 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:00 smithi029 ceph-mon[27077]: Deploying daemon osd.6 on smithi029 2024-02-10T06:42:01.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:00 smithi008 ceph-mon[24646]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T06:42:01.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:00 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T06:42:01.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:00 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:42:01.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:00 smithi008 ceph-mon[24646]: Deploying daemon osd.6 on smithi029 2024-02-10T06:42:03.139 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:02 smithi029 ceph-mon[27077]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T06:42:03.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:02 smithi008 ceph-mon[24646]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T06:42:05.089 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:04 smithi029 ceph-mon[27077]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T06:42:05.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:04 smithi008 ceph-mon[24646]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T06:42:06.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:06 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:06.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:06 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:42:06.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:06 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:42:06.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:06 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:42:06.881 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:06 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:06.881 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:06 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:42:06.881 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:06 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:42:06.881 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:06 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:42:07.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:07 smithi008 ceph-mon[24646]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T06:42:07.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:07 smithi029 ceph-mon[27077]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T06:42:08.403 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 6 on host 'smithi029' 2024-02-10T06:42:09.388 DEBUG:teuthology.orchestra.run.smithi029:osd.6> sudo journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.6.service 2024-02-10T06:42:09.390 INFO:tasks.cephadm:Deploying osd.7 on smithi029 with /dev/vg_nvme/lv_1... 2024-02-10T06:42:09.391 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-10T06:42:09.683 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:09 smithi029 ceph-mon[27077]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T06:42:09.683 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:09 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:09.684 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:09 smithi029 ceph-mon[27077]: from='osd.6 [v2:172.21.15.29:6816/3176779094,v1:172.21.15.29:6817/3176779094]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T06:42:09.684 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:09 smithi029 ceph-mon[27077]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T06:42:09.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:09 smithi008 ceph-mon[24646]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T06:42:09.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:09 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:09.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:09 smithi008 ceph-mon[24646]: from='osd.6 [v2:172.21.15.29:6816/3176779094,v1:172.21.15.29:6817/3176779094]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T06:42:09.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:09 smithi008 ceph-mon[24646]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T06:42:10.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:10 smithi008 ceph-mon[24646]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T06:42:10.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:10 smithi008 ceph-mon[24646]: osdmap e46: 7 total, 6 up, 7 in 2024-02-10T06:42:10.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:10 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:42:10.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:10 smithi008 ceph-mon[24646]: from='osd.6 [v2:172.21.15.29:6816/3176779094,v1:172.21.15.29:6817/3176779094]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:42:10.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:10 smithi008 ceph-mon[24646]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:42:10.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:10 smithi029 ceph-mon[27077]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T06:42:10.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:10 smithi029 ceph-mon[27077]: osdmap e46: 7 total, 6 up, 7 in 2024-02-10T06:42:10.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:10 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:42:10.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:10 smithi029 ceph-mon[27077]: from='osd.6 [v2:172.21.15.29:6816/3176779094,v1:172.21.15.29:6817/3176779094]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:42:10.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:10 smithi029 ceph-mon[27077]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:42:10.959 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:42:10 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[37262]: 2024-02-10T06:42:10.494+0000 7f3a97c37700 -1 osd.6 0 waiting for initial osdmap 2024-02-10T06:42:10.959 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:42:10 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[37262]: 2024-02-10T06:42:10.502+0000 7f3a8fdc9700 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:42:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:11 smithi008 ceph-mon[24646]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T06:42:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:11 smithi008 ceph-mon[24646]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-02-10T06:42:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:11 smithi008 ceph-mon[24646]: osdmap e47: 7 total, 6 up, 7 in 2024-02-10T06:42:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:11 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:42:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:11 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:42:11.859 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:11 smithi029 ceph-mon[27077]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T06:42:11.859 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:11 smithi029 ceph-mon[27077]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-02-10T06:42:11.860 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:11 smithi029 ceph-mon[27077]: osdmap e47: 7 total, 6 up, 7 in 2024-02-10T06:42:11.860 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:11 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:42:11.860 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:11 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:42:12.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:12 smithi029 ceph-mon[27077]: purged_snaps scrub starts 2024-02-10T06:42:12.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:12 smithi029 ceph-mon[27077]: purged_snaps scrub ok 2024-02-10T06:42:12.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:12 smithi029 ceph-mon[27077]: osd.6 [v2:172.21.15.29:6816/3176779094,v1:172.21.15.29:6817/3176779094] boot 2024-02-10T06:42:12.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:12 smithi029 ceph-mon[27077]: osdmap e48: 7 total, 7 up, 7 in 2024-02-10T06:42:12.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:12 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:42:12.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:12 smithi008 ceph-mon[24646]: purged_snaps scrub starts 2024-02-10T06:42:12.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:12 smithi008 ceph-mon[24646]: purged_snaps scrub ok 2024-02-10T06:42:12.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:12 smithi008 ceph-mon[24646]: osd.6 [v2:172.21.15.29:6816/3176779094,v1:172.21.15.29:6817/3176779094] boot 2024-02-10T06:42:12.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:12 smithi008 ceph-mon[24646]: osdmap e48: 7 total, 7 up, 7 in 2024-02-10T06:42:12.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:12 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:42:13.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:13 smithi008 ceph-mon[24646]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:13.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:13 smithi008 ceph-mon[24646]: osdmap e49: 7 total, 7 up, 7 in 2024-02-10T06:42:13.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:13 smithi029 ceph-mon[27077]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:13.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:13 smithi029 ceph-mon[27077]: osdmap e49: 7 total, 7 up, 7 in 2024-02-10T06:42:14.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:14 smithi008 ceph-mon[24646]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T06:42:14.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:14 smithi008 ceph-mon[24646]: osdmap e51: 7 total, 7 up, 7 in 2024-02-10T06:42:14.811 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:14 smithi029 ceph-mon[27077]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T06:42:14.811 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:14 smithi029 ceph-mon[27077]: osdmap e51: 7 total, 7 up, 7 in 2024-02-10T06:42:15.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:15 smithi008 ceph-mon[24646]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:15.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:15 smithi008 ceph-mon[24646]: osdmap e52: 7 total, 7 up, 7 in 2024-02-10T06:42:15.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:15 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:15.871 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:15 smithi029 ceph-mon[27077]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:15.871 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:15 smithi029 ceph-mon[27077]: osdmap e52: 7 total, 7 up, 7 in 2024-02-10T06:42:15.871 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:15 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:17.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:17 smithi029 ceph-mon[27077]: pgmap v156: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:17.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:17 smithi008 ceph-mon[24646]: pgmap v156: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:18.054 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-10T06:42:18.054 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-10T06:42:18.054 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-02-10T06:42:18.054 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-02-10T06:42:18.054 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00821015 s, 1.3 GB/s 2024-02-10T06:42:18.054 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-02-10T06:42:18.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_1 2024-02-10T06:42:19.636 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:19 smithi029 ceph-mon[27077]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T06:42:19.636 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:19 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:19.636 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:19 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:42:19.636 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:19 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:42:19.636 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:19 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:42:19.636 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:19 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:19.636 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:19 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:19.637 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:19 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:42:19.637 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:19 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:42:19.637 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:19 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:42:19.734 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:19 smithi008 ceph-mon[24646]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T06:42:19.735 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:19 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:19.735 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:19 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:42:19.735 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:19 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:42:19.735 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:19 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:42:19.735 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:19 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:19.735 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:19 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:19.735 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:19 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:42:19.735 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:19 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:42:19.735 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:19 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:42:20.862 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:20 smithi029 ceph-mon[27077]: Detected new or changed devices on smithi029 2024-02-10T06:42:20.862 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:20 smithi029 ceph-mon[27077]: Adjusting osd_memory_target on smithi029 to 5313M 2024-02-10T06:42:20.862 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:20 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:21.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:20 smithi008 ceph-mon[24646]: Detected new or changed devices on smithi029 2024-02-10T06:42:21.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:20 smithi008 ceph-mon[24646]: Adjusting osd_memory_target on smithi029 to 5313M 2024-02-10T06:42:21.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:20 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:21.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:21 smithi029 ceph-mon[27077]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-02-10T06:42:21.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:21 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:42:21.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:21 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:42:21.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:21 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:42:22.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:21 smithi008 ceph-mon[24646]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-02-10T06:42:22.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:21 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:42:22.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:21 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:42:22.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:21 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:42:22.949 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:22 smithi029 ceph-mon[27077]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:42:23.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:22 smithi008 ceph-mon[24646]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:42:24.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:23 smithi008 ceph-mon[24646]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-02-10T06:42:24.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:23 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/3632805767' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec5faad3-a87e-482e-a788-153765d3c44a"}]: dispatch 2024-02-10T06:42:24.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:23 smithi008 ceph-mon[24646]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec5faad3-a87e-482e-a788-153765d3c44a"}]: dispatch 2024-02-10T06:42:24.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:23 smithi008 ceph-mon[24646]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec5faad3-a87e-482e-a788-153765d3c44a"}]': finished 2024-02-10T06:42:24.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:23 smithi008 ceph-mon[24646]: osdmap e53: 8 total, 7 up, 8 in 2024-02-10T06:42:24.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:42:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:23 smithi029 ceph-mon[27077]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-02-10T06:42:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:23 smithi029 ceph-mon[27077]: from='client.? 172.21.15.29:0/3632805767' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec5faad3-a87e-482e-a788-153765d3c44a"}]: dispatch 2024-02-10T06:42:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:23 smithi029 ceph-mon[27077]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec5faad3-a87e-482e-a788-153765d3c44a"}]: dispatch 2024-02-10T06:42:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:23 smithi029 ceph-mon[27077]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec5faad3-a87e-482e-a788-153765d3c44a"}]': finished 2024-02-10T06:42:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:23 smithi029 ceph-mon[27077]: osdmap e53: 8 total, 7 up, 8 in 2024-02-10T06:42:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:42:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:24 smithi029 ceph-mon[27077]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-02-10T06:42:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:42:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:42:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:24 smithi029 ceph-mon[27077]: from='client.? 172.21.15.29:0/2302009944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:42:25.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:24 smithi008 ceph-mon[24646]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-02-10T06:42:25.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:42:25.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:42:25.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:24 smithi008 ceph-mon[24646]: from='client.? 172.21.15.29:0/2302009944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:42:27.138 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:26 smithi029 ceph-mon[27077]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T06:42:27.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:26 smithi008 ceph-mon[24646]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T06:42:29.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:28 smithi029 ceph-mon[27077]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:29.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:28 smithi008 ceph-mon[24646]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:30 smithi029 ceph-mon[27077]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:30 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T06:42:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:30 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:42:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:30 smithi029 ceph-mon[27077]: Deploying daemon osd.7 on smithi029 2024-02-10T06:42:31.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:30 smithi008 ceph-mon[24646]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:31.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:30 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T06:42:31.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:30 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:42:31.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:30 smithi008 ceph-mon[24646]: Deploying daemon osd.7 on smithi029 2024-02-10T06:42:33.166 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:32 smithi029 ceph-mon[27077]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:33.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:32 smithi008 ceph-mon[24646]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:35.096 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:34 smithi029 ceph-mon[27077]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:35.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:34 smithi008 ceph-mon[24646]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:36.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:36 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:36.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:36 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:42:36.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:36 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:42:36.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:36 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:42:36.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:36 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:36.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:36 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:42:36.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:36 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:42:36.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:36 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:42:37.624 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:37 smithi029 ceph-mon[27077]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:37.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:37 smithi008 ceph-mon[24646]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:38.635 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 7 on host 'smithi029' 2024-02-10T06:42:39.301 DEBUG:teuthology.orchestra.run.smithi029:osd.7> sudo journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.7.service 2024-02-10T06:42:39.303 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-10T06:42:39.303 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-10T06:42:39.583 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:39 smithi029 ceph-mon[27077]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:39.583 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:39 smithi029 ceph-mon[27077]: from='osd.7 [v2:172.21.15.29:6824/1694099296,v1:172.21.15.29:6825/1694099296]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T06:42:39.583 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:39 smithi029 ceph-mon[27077]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T06:42:39.583 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:39 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:39.624 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:39 smithi008 ceph-mon[24646]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:39.624 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:39 smithi008 ceph-mon[24646]: from='osd.7 [v2:172.21.15.29:6824/1694099296,v1:172.21.15.29:6825/1694099296]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T06:42:39.624 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:39 smithi008 ceph-mon[24646]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T06:42:39.624 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:39 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:40.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:40 smithi029 ceph-mon[27077]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T06:42:40.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:40 smithi029 ceph-mon[27077]: osdmap e54: 8 total, 7 up, 8 in 2024-02-10T06:42:40.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:40 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:42:40.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:40 smithi029 ceph-mon[27077]: from='osd.7 [v2:172.21.15.29:6824/1694099296,v1:172.21.15.29:6825/1694099296]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:42:40.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:40 smithi029 ceph-mon[27077]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:42:40.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:40 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:40.459 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:42:40 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[40346]: 2024-02-10T06:42:40.382+0000 7f22f293e700 -1 osd.7 0 waiting for initial osdmap 2024-02-10T06:42:40.459 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:42:40 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[40346]: 2024-02-10T06:42:40.389+0000 7f22ed2d5700 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:42:40.638 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:40 smithi008 ceph-mon[24646]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T06:42:40.639 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:40 smithi008 ceph-mon[24646]: osdmap e54: 8 total, 7 up, 8 in 2024-02-10T06:42:40.639 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:40 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:42:40.639 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:40 smithi008 ceph-mon[24646]: from='osd.7 [v2:172.21.15.29:6824/1694099296,v1:172.21.15.29:6825/1694099296]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:42:40.639 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:40 smithi008 ceph-mon[24646]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:42:40.639 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:40 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:41.353 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:42:41.614 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:41 smithi008 ceph-mon[24646]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:41.614 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:41 smithi008 ceph-mon[24646]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-02-10T06:42:41.614 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:41 smithi008 ceph-mon[24646]: osdmap e55: 8 total, 7 up, 8 in 2024-02-10T06:42:41.614 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:41 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:42:41.615 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:41 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:42:41.615 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:41 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/2580244491' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T06:42:41.675 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:41 smithi029 ceph-mon[27077]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T06:42:41.675 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:41 smithi029 ceph-mon[27077]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-02-10T06:42:41.675 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:41 smithi029 ceph-mon[27077]: osdmap e55: 8 total, 7 up, 8 in 2024-02-10T06:42:41.675 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:41 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:42:41.675 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:41 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:42:41.675 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:41 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/2580244491' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T06:42:42.084 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":55,"num_osds":8,"num_up_osds":7,"osd_up_since":1707547331,"num_in_osds":8,"osd_in_since":1707547343,"num_remapped_pgs":0} 2024-02-10T06:42:42.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:42 smithi029 ceph-mon[27077]: purged_snaps scrub starts 2024-02-10T06:42:42.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:42 smithi029 ceph-mon[27077]: purged_snaps scrub ok 2024-02-10T06:42:42.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:42 smithi029 ceph-mon[27077]: osd.7 [v2:172.21.15.29:6824/1694099296,v1:172.21.15.29:6825/1694099296] boot 2024-02-10T06:42:42.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:42 smithi029 ceph-mon[27077]: osdmap e56: 8 total, 8 up, 8 in 2024-02-10T06:42:42.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:42 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:42:42.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:42 smithi029 ceph-mon[27077]: osdmap e57: 8 total, 8 up, 8 in 2024-02-10T06:42:42.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:42 smithi008 ceph-mon[24646]: purged_snaps scrub starts 2024-02-10T06:42:42.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:42 smithi008 ceph-mon[24646]: purged_snaps scrub ok 2024-02-10T06:42:42.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:42 smithi008 ceph-mon[24646]: osd.7 [v2:172.21.15.29:6824/1694099296,v1:172.21.15.29:6825/1694099296] boot 2024-02-10T06:42:42.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:42 smithi008 ceph-mon[24646]: osdmap e56: 8 total, 8 up, 8 in 2024-02-10T06:42:42.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:42 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:42:42.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:42 smithi008 ceph-mon[24646]: osdmap e57: 8 total, 8 up, 8 in 2024-02-10T06:42:43.085 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-10T06:42:43.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:43 smithi029 ceph-mon[27077]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:42:43.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:43 smithi008 ceph-mon[24646]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:42:44.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:44 smithi008 ceph-mon[24646]: osdmap e58: 8 total, 8 up, 8 in 2024-02-10T06:42:44.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:44 smithi008 ceph-mon[24646]: Detected new or changed devices on smithi029 2024-02-10T06:42:44.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:44.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:42:44.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:42:44.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:42:44.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:42:44.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:44 smithi008 ceph-mon[24646]: Adjusting osd_memory_target on smithi029 to 3985M 2024-02-10T06:42:44.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:44.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:44.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:42:44.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:42:44.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:42:44.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:44 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:44.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:44 smithi008 ceph-mon[24646]: osdmap e59: 8 total, 8 up, 8 in 2024-02-10T06:42:44.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:44 smithi029 ceph-mon[27077]: osdmap e58: 8 total, 8 up, 8 in 2024-02-10T06:42:44.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:44 smithi029 ceph-mon[27077]: Detected new or changed devices on smithi029 2024-02-10T06:42:44.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:44.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:42:44.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:42:44.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:42:44.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:42:44.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:44 smithi029 ceph-mon[27077]: Adjusting osd_memory_target on smithi029 to 3985M 2024-02-10T06:42:44.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:44.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:44.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:42:44.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:42:44.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:42:44.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:44 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:42:44.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:44 smithi029 ceph-mon[27077]: osdmap e59: 8 total, 8 up, 8 in 2024-02-10T06:42:45.182 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:42:45.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:45 smithi029 ceph-mon[27077]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:42:45.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:45 smithi029 ceph-mon[27077]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T06:42:45.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:45 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/2680448704' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T06:42:45.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:45 smithi008 ceph-mon[24646]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:42:45.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:45 smithi008 ceph-mon[24646]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T06:42:45.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:45 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/2680448704' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T06:42:46.029 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":60,"num_osds":8,"num_up_osds":8,"osd_up_since":1707547361,"num_in_osds":8,"osd_in_since":1707547343,"num_remapped_pgs":0} 2024-02-10T06:42:46.030 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T06:42:46.253 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:42:47.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:47 smithi029 ceph-mon[27077]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:42:47.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:47 smithi008 ceph-mon[24646]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:42:48.220 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:42:48.221 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":60,"fsid":"a5f941ec-c7de-11ee-95b8-87774f69a715","created":"2024-02-10T06:36:29.227934+0000","modified":"2024-02-10T06:42:44.801027+0000","last_up_change":"2024-02-10T06:42:41.376690+0000","last_in_change":"2024-02-10T06:42:23.666176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T06:40:32.189361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ff16b296-565e-40ff-b128-8f652db427d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":903496854},{"type":"v1","addr":"172.21.15.8:6803","nonce":903496854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":903496854},{"type":"v1","addr":"172.21.15.8:6805","nonce":903496854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":903496854},{"type":"v1","addr":"172.21.15.8:6809","nonce":903496854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":903496854},{"type":"v1","addr":"172.21.15.8:6807","nonce":903496854}]},"public_addr":"172.21.15.8:6803/903496854","cluster_addr":"172.21.15.8:6805/903496854","heartbeat_back_addr":"172.21.15.8:6809/903496854","heartbeat_front_addr":"172.21.15.8:6807/903496854","state":["exists","up"]},{"osd":1,"uuid":"b2869b5b-c167-4772-93fa-a2b817026314","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":3571451027},{"type":"v1","addr":"172.21.15.8:6811","nonce":3571451027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":3571451027},{"type":"v1","addr":"172.21.15.8:6813","nonce":3571451027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":3571451027},{"type":"v1","addr":"172.21.15.8:6817","nonce":3571451027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":3571451027},{"type":"v1","addr":"172.21.15.8:6815","nonce":3571451027}]},"public_addr":"172.21.15.8:6811/3571451027","cluster_addr":"172.21.15.8:6813/3571451027","heartbeat_back_addr":"172.21.15.8:6817/3571451027","heartbeat_front_addr":"172.21.15.8:6815/3571451027","state":["exists","up"]},{"osd":2,"uuid":"ffae79d2-47bf-45d3-8d66-0cc121397b01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":2223271854},{"type":"v1","addr":"172.21.15.8:6819","nonce":2223271854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":2223271854},{"type":"v1","addr":"172.21.15.8:6821","nonce":2223271854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":2223271854},{"type":"v1","addr":"172.21.15.8:6825","nonce":2223271854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":2223271854},{"type":"v1","addr":"172.21.15.8:6823","nonce":2223271854}]},"public_addr":"172.21.15.8:6819/2223271854","cluster_addr":"172.21.15.8:6821/2223271854","heartbeat_back_addr":"172.21.15.8:6825/2223271854","heartbeat_front_addr":"172.21.15.8:6823/2223271854","state":["exists","up"]},{"osd":3,"uuid":"a73469d1-315f-41d5-81d1-b9c69df34e8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":1037657938},{"type":"v1","addr":"172.21.15.8:6827","nonce":1037657938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":1037657938},{"type":"v1","addr":"172.21.15.8:6829","nonce":1037657938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6832","nonce":1037657938},{"type":"v1","addr":"172.21.15.8:6833","nonce":1037657938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":1037657938},{"type":"v1","addr":"172.21.15.8:6831","nonce":1037657938}]},"public_addr":"172.21.15.8:6827/1037657938","cluster_addr":"172.21.15.8:6829/1037657938","heartbeat_back_addr":"172.21.15.8:6833/1037657938","heartbeat_front_addr":"172.21.15.8:6831/1037657938","state":["exists","up"]},{"osd":4,"uuid":"1568e47a-0969-40b2-bf29-8a1ccc4d5752","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3369615219},{"type":"v1","addr":"172.21.15.29:6801","nonce":3369615219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3369615219},{"type":"v1","addr":"172.21.15.29:6803","nonce":3369615219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3369615219},{"type":"v1","addr":"172.21.15.29:6807","nonce":3369615219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3369615219},{"type":"v1","addr":"172.21.15.29:6805","nonce":3369615219}]},"public_addr":"172.21.15.29:6801/3369615219","cluster_addr":"172.21.15.29:6803/3369615219","heartbeat_back_addr":"172.21.15.29:6807/3369615219","heartbeat_front_addr":"172.21.15.29:6805/3369615219","state":["exists","up"]},{"osd":5,"uuid":"91c1b854-0a37-4643-9a32-0307c6c1bfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":4103839609},{"type":"v1","addr":"172.21.15.29:6809","nonce":4103839609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":4103839609},{"type":"v1","addr":"172.21.15.29:6811","nonce":4103839609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":4103839609},{"type":"v1","addr":"172.21.15.29:6815","nonce":4103839609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4103839609},{"type":"v1","addr":"172.21.15.29:6813","nonce":4103839609}]},"public_addr":"172.21.15.29:6809/4103839609","cluster_addr":"172.21.15.29:6811/4103839609","heartbeat_back_addr":"172.21.15.29:6815/4103839609","heartbeat_front_addr":"172.21.15.29:6813/4103839609","state":["exists","up"]},{"osd":6,"uuid":"d24e3ca9-b39f-4e81-855e-b3305151ad5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3176779094},{"type":"v1","addr":"172.21.15.29:6817","nonce":3176779094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3176779094},{"type":"v1","addr":"172.21.15.29:6819","nonce":3176779094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3176779094},{"type":"v1","addr":"172.21.15.29:6823","nonce":3176779094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3176779094},{"type":"v1","addr":"172.21.15.29:6821","nonce":3176779094}]},"public_addr":"172.21.15.29:6817/3176779094","cluster_addr":"172.21.15.29:6819/3176779094","heartbeat_back_addr":"172.21.15.29:6823/3176779094","heartbeat_front_addr":"172.21.15.29:6821/3176779094","state":["exists","up"]},{"osd":7,"uuid":"ec5faad3-a87e-482e-a788-153765d3c44a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1694099296},{"type":"v1","addr":"172.21.15.29:6825","nonce":1694099296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1694099296},{"type":"v1","addr":"172.21.15.29:6827","nonce":1694099296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1694099296},{"type":"v1","addr":"172.21.15.29:6831","nonce":1694099296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1694099296},{"type":"v1","addr":"172.21.15.29:6829","nonce":1694099296}]},"public_addr":"172.21.15.29:6825/1694099296","cluster_addr":"172.21.15.29:6827/1694099296","heartbeat_back_addr":"172.21.15.29:6831/1694099296","heartbeat_front_addr":"172.21.15.29:6829/1694099296","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:39:30.151630+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:39:59.635569+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:40:28.379690+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:40:56.392835+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:41:18.916847+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:41:44.275971+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:42:09.560998+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:42:39.489162+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.8:0/4271303424":"2024-02-11T06:38:23.637113+0000","172.21.15.8:0/1375520004":"2024-02-11T06:38:23.637113+0000","172.21.15.8:0/1307806439":"2024-02-11T06:37:30.756336+0000","172.21.15.8:0/295422627":"2024-02-11T06:38:23.637113+0000","172.21.15.8:0/1181547877":"2024-02-11T06:37:30.756336+0000","172.21.15.8:6800/4241104945":"2024-02-11T06:38:23.637113+0000","172.21.15.8:6800/1272476176":"2024-02-11T06:37:30.756336+0000","172.21.15.8:0/3988115812":"2024-02-11T06:37:30.756336+0000","172.21.15.8:0/1774202401":"2024-02-11T06:37:00.743293+0000","172.21.15.8:6801/3386597727":"2024-02-11T06:37:00.743293+0000","172.21.15.8:6801/1272476176":"2024-02-11T06:37:30.756336+0000","172.21.15.8:0/3108325203":"2024-02-11T06:37:00.743293+0000","172.21.15.8:6800/3386597727":"2024-02-11T06:37:00.743293+0000","172.21.15.8:6801/4241104945":"2024-02-11T06:38:23.637113+0000","172.21.15.8:0/145175684":"2024-02-11T06:37:00.743293+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T06:42:48.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:48 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/4274611662' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:42:48.722 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:48 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/4274611662' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:42:48.791 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-10T06:40:32.189361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-02-10T06:42:48.791 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-10T06:42:49.010 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:42:49.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:49 smithi029 ceph-mon[27077]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-02-10T06:42:49.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:49 smithi008 ceph-mon[24646]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-02-10T06:42:51.032 INFO:teuthology.orchestra.run.smithi008.stdout:pg_num: 1 2024-02-10T06:42:51.589 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:51 smithi008 ceph-mon[24646]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-02-10T06:42:51.589 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:51 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/714101955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T06:42:51.613 INFO:tasks.cephadm:Setting up client nodes... 2024-02-10T06:42:51.613 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T06:42:51.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:51 smithi029 ceph-mon[27077]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-02-10T06:42:51.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:51 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/714101955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T06:42:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:53 smithi029 ceph-mon[27077]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-02-10T06:42:53.764 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:53 smithi008 ceph-mon[24646]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-02-10T06:42:53.830 INFO:teuthology.orchestra.run.smithi008.stdout:[client.0] 2024-02-10T06:42:53.831 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQDtGsdlqKpHMRAAIcmrWu8faR8xwNVIELxytA== 2024-02-10T06:42:54.520 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T06:42:54.520 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-10T06:42:54.520 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-10T06:42:54.563 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-10T06:42:54.563 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-10T06:42:54.563 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph mgr dump --format=json 2024-02-10T06:42:54.820 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:42:55.067 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:54 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1362673393' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:42:55.067 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:54 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1362673393' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T06:42:55.067 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:54 smithi008 ceph-mon[24646]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T06:42:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:54 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1362673393' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:42:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:54 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1362673393' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T06:42:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:54 smithi029 ceph-mon[27077]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T06:42:56.726 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:42:57.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:56 smithi008 ceph-mon[24646]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-02-10T06:42:57.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:56 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1230861854' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T06:42:57.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:56 smithi029 ceph-mon[27077]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-02-10T06:42:57.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:56 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1230861854' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T06:42:57.445 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":19,"active_gid":14202,"active_name":"smithi008.pxifbt","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":815256931},{"type":"v1","addr":"172.21.15.8:6801","nonce":815256931}]},"active_addr":"172.21.15.8:6801/815256931","active_change":"2024-02-10T06:38:23.637408+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14224,"name":"smithi029.fuvrtf","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.8:8443/","prometheus":"http://172.21.15.8:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":2923124677}]},{"addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":621617966}]},{"addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":892627382}]},{"addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":3420102544}]}]}} 2024-02-10T06:42:57.450 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-10T06:42:57.450 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-10T06:42:57.450 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T06:42:57.669 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:42:59.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:58 smithi008 ceph-mon[24646]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-02-10T06:42:59.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:58 smithi029 ceph-mon[27077]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-02-10T06:42:59.489 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:42:59.489 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":60,"fsid":"a5f941ec-c7de-11ee-95b8-87774f69a715","created":"2024-02-10T06:36:29.227934+0000","modified":"2024-02-10T06:42:44.801027+0000","last_up_change":"2024-02-10T06:42:41.376690+0000","last_in_change":"2024-02-10T06:42:23.666176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T06:40:32.189361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ff16b296-565e-40ff-b128-8f652db427d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":903496854},{"type":"v1","addr":"172.21.15.8:6803","nonce":903496854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":903496854},{"type":"v1","addr":"172.21.15.8:6805","nonce":903496854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":903496854},{"type":"v1","addr":"172.21.15.8:6809","nonce":903496854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":903496854},{"type":"v1","addr":"172.21.15.8:6807","nonce":903496854}]},"public_addr":"172.21.15.8:6803/903496854","cluster_addr":"172.21.15.8:6805/903496854","heartbeat_back_addr":"172.21.15.8:6809/903496854","heartbeat_front_addr":"172.21.15.8:6807/903496854","state":["exists","up"]},{"osd":1,"uuid":"b2869b5b-c167-4772-93fa-a2b817026314","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":3571451027},{"type":"v1","addr":"172.21.15.8:6811","nonce":3571451027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":3571451027},{"type":"v1","addr":"172.21.15.8:6813","nonce":3571451027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":3571451027},{"type":"v1","addr":"172.21.15.8:6817","nonce":3571451027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":3571451027},{"type":"v1","addr":"172.21.15.8:6815","nonce":3571451027}]},"public_addr":"172.21.15.8:6811/3571451027","cluster_addr":"172.21.15.8:6813/3571451027","heartbeat_back_addr":"172.21.15.8:6817/3571451027","heartbeat_front_addr":"172.21.15.8:6815/3571451027","state":["exists","up"]},{"osd":2,"uuid":"ffae79d2-47bf-45d3-8d66-0cc121397b01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":2223271854},{"type":"v1","addr":"172.21.15.8:6819","nonce":2223271854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":2223271854},{"type":"v1","addr":"172.21.15.8:6821","nonce":2223271854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":2223271854},{"type":"v1","addr":"172.21.15.8:6825","nonce":2223271854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":2223271854},{"type":"v1","addr":"172.21.15.8:6823","nonce":2223271854}]},"public_addr":"172.21.15.8:6819/2223271854","cluster_addr":"172.21.15.8:6821/2223271854","heartbeat_back_addr":"172.21.15.8:6825/2223271854","heartbeat_front_addr":"172.21.15.8:6823/2223271854","state":["exists","up"]},{"osd":3,"uuid":"a73469d1-315f-41d5-81d1-b9c69df34e8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":1037657938},{"type":"v1","addr":"172.21.15.8:6827","nonce":1037657938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":1037657938},{"type":"v1","addr":"172.21.15.8:6829","nonce":1037657938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6832","nonce":1037657938},{"type":"v1","addr":"172.21.15.8:6833","nonce":1037657938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":1037657938},{"type":"v1","addr":"172.21.15.8:6831","nonce":1037657938}]},"public_addr":"172.21.15.8:6827/1037657938","cluster_addr":"172.21.15.8:6829/1037657938","heartbeat_back_addr":"172.21.15.8:6833/1037657938","heartbeat_front_addr":"172.21.15.8:6831/1037657938","state":["exists","up"]},{"osd":4,"uuid":"1568e47a-0969-40b2-bf29-8a1ccc4d5752","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3369615219},{"type":"v1","addr":"172.21.15.29:6801","nonce":3369615219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3369615219},{"type":"v1","addr":"172.21.15.29:6803","nonce":3369615219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3369615219},{"type":"v1","addr":"172.21.15.29:6807","nonce":3369615219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3369615219},{"type":"v1","addr":"172.21.15.29:6805","nonce":3369615219}]},"public_addr":"172.21.15.29:6801/3369615219","cluster_addr":"172.21.15.29:6803/3369615219","heartbeat_back_addr":"172.21.15.29:6807/3369615219","heartbeat_front_addr":"172.21.15.29:6805/3369615219","state":["exists","up"]},{"osd":5,"uuid":"91c1b854-0a37-4643-9a32-0307c6c1bfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":4103839609},{"type":"v1","addr":"172.21.15.29:6809","nonce":4103839609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":4103839609},{"type":"v1","addr":"172.21.15.29:6811","nonce":4103839609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":4103839609},{"type":"v1","addr":"172.21.15.29:6815","nonce":4103839609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4103839609},{"type":"v1","addr":"172.21.15.29:6813","nonce":4103839609}]},"public_addr":"172.21.15.29:6809/4103839609","cluster_addr":"172.21.15.29:6811/4103839609","heartbeat_back_addr":"172.21.15.29:6815/4103839609","heartbeat_front_addr":"172.21.15.29:6813/4103839609","state":["exists","up"]},{"osd":6,"uuid":"d24e3ca9-b39f-4e81-855e-b3305151ad5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3176779094},{"type":"v1","addr":"172.21.15.29:6817","nonce":3176779094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3176779094},{"type":"v1","addr":"172.21.15.29:6819","nonce":3176779094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3176779094},{"type":"v1","addr":"172.21.15.29:6823","nonce":3176779094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3176779094},{"type":"v1","addr":"172.21.15.29:6821","nonce":3176779094}]},"public_addr":"172.21.15.29:6817/3176779094","cluster_addr":"172.21.15.29:6819/3176779094","heartbeat_back_addr":"172.21.15.29:6823/3176779094","heartbeat_front_addr":"172.21.15.29:6821/3176779094","state":["exists","up"]},{"osd":7,"uuid":"ec5faad3-a87e-482e-a788-153765d3c44a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1694099296},{"type":"v1","addr":"172.21.15.29:6825","nonce":1694099296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1694099296},{"type":"v1","addr":"172.21.15.29:6827","nonce":1694099296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1694099296},{"type":"v1","addr":"172.21.15.29:6831","nonce":1694099296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1694099296},{"type":"v1","addr":"172.21.15.29:6829","nonce":1694099296}]},"public_addr":"172.21.15.29:6825/1694099296","cluster_addr":"172.21.15.29:6827/1694099296","heartbeat_back_addr":"172.21.15.29:6831/1694099296","heartbeat_front_addr":"172.21.15.29:6829/1694099296","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:39:30.151630+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:39:59.635569+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:40:28.379690+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:40:56.392835+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:41:18.916847+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:41:44.275971+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:42:09.560998+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:42:39.489162+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.8:0/4271303424":"2024-02-11T06:38:23.637113+0000","172.21.15.8:0/1375520004":"2024-02-11T06:38:23.637113+0000","172.21.15.8:0/1307806439":"2024-02-11T06:37:30.756336+0000","172.21.15.8:0/295422627":"2024-02-11T06:38:23.637113+0000","172.21.15.8:0/1181547877":"2024-02-11T06:37:30.756336+0000","172.21.15.8:6800/4241104945":"2024-02-11T06:38:23.637113+0000","172.21.15.8:6800/1272476176":"2024-02-11T06:37:30.756336+0000","172.21.15.8:0/3988115812":"2024-02-11T06:37:30.756336+0000","172.21.15.8:0/1774202401":"2024-02-11T06:37:00.743293+0000","172.21.15.8:6801/3386597727":"2024-02-11T06:37:00.743293+0000","172.21.15.8:6801/1272476176":"2024-02-11T06:37:30.756336+0000","172.21.15.8:0/3108325203":"2024-02-11T06:37:00.743293+0000","172.21.15.8:6800/3386597727":"2024-02-11T06:37:00.743293+0000","172.21.15.8:6801/4241104945":"2024-02-11T06:38:23.637113+0000","172.21.15.8:0/145175684":"2024-02-11T06:37:00.743293+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T06:43:00.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:42:59 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/2459712721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:43:00.346 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-10T06:43:00.346 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T06:43:00.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:42:59 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/2459712721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:43:00.570 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:01.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:00 smithi008 ceph-mon[24646]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:01.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:00 smithi029 ceph-mon[27077]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:02.344 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:43:02.344 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":60,"fsid":"a5f941ec-c7de-11ee-95b8-87774f69a715","created":"2024-02-10T06:36:29.227934+0000","modified":"2024-02-10T06:42:44.801027+0000","last_up_change":"2024-02-10T06:42:41.376690+0000","last_in_change":"2024-02-10T06:42:23.666176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T06:40:32.189361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ff16b296-565e-40ff-b128-8f652db427d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":903496854},{"type":"v1","addr":"172.21.15.8:6803","nonce":903496854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":903496854},{"type":"v1","addr":"172.21.15.8:6805","nonce":903496854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":903496854},{"type":"v1","addr":"172.21.15.8:6809","nonce":903496854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":903496854},{"type":"v1","addr":"172.21.15.8:6807","nonce":903496854}]},"public_addr":"172.21.15.8:6803/903496854","cluster_addr":"172.21.15.8:6805/903496854","heartbeat_back_addr":"172.21.15.8:6809/903496854","heartbeat_front_addr":"172.21.15.8:6807/903496854","state":["exists","up"]},{"osd":1,"uuid":"b2869b5b-c167-4772-93fa-a2b817026314","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":3571451027},{"type":"v1","addr":"172.21.15.8:6811","nonce":3571451027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":3571451027},{"type":"v1","addr":"172.21.15.8:6813","nonce":3571451027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":3571451027},{"type":"v1","addr":"172.21.15.8:6817","nonce":3571451027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":3571451027},{"type":"v1","addr":"172.21.15.8:6815","nonce":3571451027}]},"public_addr":"172.21.15.8:6811/3571451027","cluster_addr":"172.21.15.8:6813/3571451027","heartbeat_back_addr":"172.21.15.8:6817/3571451027","heartbeat_front_addr":"172.21.15.8:6815/3571451027","state":["exists","up"]},{"osd":2,"uuid":"ffae79d2-47bf-45d3-8d66-0cc121397b01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":2223271854},{"type":"v1","addr":"172.21.15.8:6819","nonce":2223271854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":2223271854},{"type":"v1","addr":"172.21.15.8:6821","nonce":2223271854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":2223271854},{"type":"v1","addr":"172.21.15.8:6825","nonce":2223271854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":2223271854},{"type":"v1","addr":"172.21.15.8:6823","nonce":2223271854}]},"public_addr":"172.21.15.8:6819/2223271854","cluster_addr":"172.21.15.8:6821/2223271854","heartbeat_back_addr":"172.21.15.8:6825/2223271854","heartbeat_front_addr":"172.21.15.8:6823/2223271854","state":["exists","up"]},{"osd":3,"uuid":"a73469d1-315f-41d5-81d1-b9c69df34e8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":1037657938},{"type":"v1","addr":"172.21.15.8:6827","nonce":1037657938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":1037657938},{"type":"v1","addr":"172.21.15.8:6829","nonce":1037657938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6832","nonce":1037657938},{"type":"v1","addr":"172.21.15.8:6833","nonce":1037657938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":1037657938},{"type":"v1","addr":"172.21.15.8:6831","nonce":1037657938}]},"public_addr":"172.21.15.8:6827/1037657938","cluster_addr":"172.21.15.8:6829/1037657938","heartbeat_back_addr":"172.21.15.8:6833/1037657938","heartbeat_front_addr":"172.21.15.8:6831/1037657938","state":["exists","up"]},{"osd":4,"uuid":"1568e47a-0969-40b2-bf29-8a1ccc4d5752","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3369615219},{"type":"v1","addr":"172.21.15.29:6801","nonce":3369615219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3369615219},{"type":"v1","addr":"172.21.15.29:6803","nonce":3369615219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3369615219},{"type":"v1","addr":"172.21.15.29:6807","nonce":3369615219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3369615219},{"type":"v1","addr":"172.21.15.29:6805","nonce":3369615219}]},"public_addr":"172.21.15.29:6801/3369615219","cluster_addr":"172.21.15.29:6803/3369615219","heartbeat_back_addr":"172.21.15.29:6807/3369615219","heartbeat_front_addr":"172.21.15.29:6805/3369615219","state":["exists","up"]},{"osd":5,"uuid":"91c1b854-0a37-4643-9a32-0307c6c1bfd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":4103839609},{"type":"v1","addr":"172.21.15.29:6809","nonce":4103839609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":4103839609},{"type":"v1","addr":"172.21.15.29:6811","nonce":4103839609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":4103839609},{"type":"v1","addr":"172.21.15.29:6815","nonce":4103839609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4103839609},{"type":"v1","addr":"172.21.15.29:6813","nonce":4103839609}]},"public_addr":"172.21.15.29:6809/4103839609","cluster_addr":"172.21.15.29:6811/4103839609","heartbeat_back_addr":"172.21.15.29:6815/4103839609","heartbeat_front_addr":"172.21.15.29:6813/4103839609","state":["exists","up"]},{"osd":6,"uuid":"d24e3ca9-b39f-4e81-855e-b3305151ad5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3176779094},{"type":"v1","addr":"172.21.15.29:6817","nonce":3176779094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3176779094},{"type":"v1","addr":"172.21.15.29:6819","nonce":3176779094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":3176779094},{"type":"v1","addr":"172.21.15.29:6823","nonce":3176779094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":3176779094},{"type":"v1","addr":"172.21.15.29:6821","nonce":3176779094}]},"public_addr":"172.21.15.29:6817/3176779094","cluster_addr":"172.21.15.29:6819/3176779094","heartbeat_back_addr":"172.21.15.29:6823/3176779094","heartbeat_front_addr":"172.21.15.29:6821/3176779094","state":["exists","up"]},{"osd":7,"uuid":"ec5faad3-a87e-482e-a788-153765d3c44a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1694099296},{"type":"v1","addr":"172.21.15.29:6825","nonce":1694099296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1694099296},{"type":"v1","addr":"172.21.15.29:6827","nonce":1694099296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1694099296},{"type":"v1","addr":"172.21.15.29:6831","nonce":1694099296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1694099296},{"type":"v1","addr":"172.21.15.29:6829","nonce":1694099296}]},"public_addr":"172.21.15.29:6825/1694099296","cluster_addr":"172.21.15.29:6827/1694099296","heartbeat_back_addr":"172.21.15.29:6831/1694099296","heartbeat_front_addr":"172.21.15.29:6829/1694099296","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:39:30.151630+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:39:59.635569+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:40:28.379690+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:40:56.392835+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:41:18.916847+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:41:44.275971+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:42:09.560998+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:42:39.489162+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.8:0/4271303424":"2024-02-11T06:38:23.637113+0000","172.21.15.8:0/1375520004":"2024-02-11T06:38:23.637113+0000","172.21.15.8:0/1307806439":"2024-02-11T06:37:30.756336+0000","172.21.15.8:0/295422627":"2024-02-11T06:38:23.637113+0000","172.21.15.8:0/1181547877":"2024-02-11T06:37:30.756336+0000","172.21.15.8:6800/4241104945":"2024-02-11T06:38:23.637113+0000","172.21.15.8:6800/1272476176":"2024-02-11T06:37:30.756336+0000","172.21.15.8:0/3988115812":"2024-02-11T06:37:30.756336+0000","172.21.15.8:0/1774202401":"2024-02-11T06:37:00.743293+0000","172.21.15.8:6801/3386597727":"2024-02-11T06:37:00.743293+0000","172.21.15.8:6801/1272476176":"2024-02-11T06:37:30.756336+0000","172.21.15.8:0/3108325203":"2024-02-11T06:37:00.743293+0000","172.21.15.8:6800/3386597727":"2024-02-11T06:37:00.743293+0000","172.21.15.8:6801/4241104945":"2024-02-11T06:38:23.637113+0000","172.21.15.8:0/145175684":"2024-02-11T06:37:00.743293+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T06:43:02.942 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-10T06:43:02.942 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-10T06:43:02.943 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-10T06:43:02.943 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-10T06:43:02.944 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-10T06:43:02.944 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-10T06:43:02.944 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-10T06:43:02.945 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-10T06:43:03.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:02 smithi008 ceph-mon[24646]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:03.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:02 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1004314735' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:43:03.394 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:03.394 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:03.394 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:03.399 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:03.399 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:03.400 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:03.400 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:03.403 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:03.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:02 smithi029 ceph-mon[27077]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:03.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:02 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1004314735' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:43:05.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:04 smithi008 ceph-mon[24646]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:05.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:04 smithi029 ceph-mon[27077]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:07.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:06 smithi029 ceph-mon[27077]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:07.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:06 smithi008 ceph-mon[24646]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:09.213 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:08 smithi008 ceph-mon[24646]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:09.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:08 smithi029 ceph-mon[27077]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:11.191 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:10 smithi008 ceph-mon[24646]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:11.260 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:10 smithi029 ceph-mon[27077]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:13.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:12 smithi008 ceph-mon[24646]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:13.322 INFO:teuthology.orchestra.run.smithi008.stdout:240518168585 2024-02-10T06:43:13.322 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-10T06:43:13.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:12 smithi029 ceph-mon[27077]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:13.973 INFO:teuthology.orchestra.run.smithi008.stdout:115964117021 2024-02-10T06:43:13.974 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-10T06:43:15.190 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:14 smithi008 ceph-mon[24646]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:15.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:14 smithi029 ceph-mon[27077]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:15.377 INFO:teuthology.orchestra.run.smithi008.stdout:137438953497 2024-02-10T06:43:15.378 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-10T06:43:15.979 INFO:teuthology.orchestra.run.smithi008.stdout:81604378659 2024-02-10T06:43:15.980 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-10T06:43:16.823 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:16.824 INFO:teuthology.orchestra.run.smithi008.stdout:206158430222 2024-02-10T06:43:16.824 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-10T06:43:17.078 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:16 smithi008 ceph-mon[24646]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:17.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:16 smithi029 ceph-mon[27077]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:18.272 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:18 smithi008 ceph-mon[24646]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:19.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:18 smithi029 ceph-mon[27077]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:19.622 INFO:teuthology.orchestra.run.smithi008.stdout:171798691861 2024-02-10T06:43:19.623 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-10T06:43:20.916 INFO:teuthology.orchestra.run.smithi008.stdout:60129542186 2024-02-10T06:43:20.916 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-10T06:43:21.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:20 smithi008 ceph-mon[24646]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:21.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:20 smithi029 ceph-mon[27077]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:22.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:21 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1648185162' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T06:43:22.056 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:22.057 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:22.057 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:21 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1648185162' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T06:43:22.528 INFO:teuthology.orchestra.run.smithi008.stdout:240518168586 2024-02-10T06:43:22.736 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:22.736 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:23.132 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:22 smithi008 ceph-mon[24646]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:23.156 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168585 got 240518168586 for osd.7 2024-02-10T06:43:23.156 DEBUG:teuthology.parallel:result is None 2024-02-10T06:43:23.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:22 smithi029 ceph-mon[27077]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:24 smithi029 ceph-mon[27077]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:43:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:43:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:24 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/922230958' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T06:43:25.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:24 smithi008 ceph-mon[24646]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:25.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:43:25.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:43:25.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:24 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/922230958' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T06:43:25.625 INFO:teuthology.orchestra.run.smithi008.stdout:115964117023 2024-02-10T06:43:26.163 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117021 got 115964117023 for osd.3 2024-02-10T06:43:26.164 DEBUG:teuthology.parallel:result is None 2024-02-10T06:43:27.201 INFO:teuthology.orchestra.run.smithi008.stdout:38654705714 2024-02-10T06:43:27.202 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-10T06:43:27.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:26 smithi008 ceph-mon[24646]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:27.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:26 smithi029 ceph-mon[27077]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:29.176 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:28 smithi008 ceph-mon[24646]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:29.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:28 smithi029 ceph-mon[27077]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:30.191 INFO:teuthology.orchestra.run.smithi008.stdout:171798691863 2024-02-10T06:43:30.377 INFO:teuthology.orchestra.run.smithi008.stdout:81604378662 2024-02-10T06:43:31.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:30 smithi008 ceph-mon[24646]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:31.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:30 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1372362330' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T06:43:31.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:30 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1742672331' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T06:43:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:30 smithi029 ceph-mon[27077]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:30 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1372362330' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T06:43:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:30 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1742672331' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T06:43:31.838 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378659 got 81604378662 for osd.2 2024-02-10T06:43:31.838 DEBUG:teuthology.parallel:result is None 2024-02-10T06:43:32.440 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691861 got 171798691863 for osd.5 2024-02-10T06:43:32.440 DEBUG:teuthology.parallel:result is None 2024-02-10T06:43:32.658 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:32.832 INFO:teuthology.orchestra.run.smithi008.stdout:206158430227 2024-02-10T06:43:33.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:32 smithi008 ceph-mon[24646]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:33.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:32 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/784540392' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T06:43:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:32 smithi029 ceph-mon[27077]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:32 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/784540392' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T06:43:33.462 INFO:teuthology.orchestra.run.smithi008.stdout:137438953500 2024-02-10T06:43:33.805 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430222 got 206158430227 for osd.6 2024-02-10T06:43:33.805 DEBUG:teuthology.parallel:result is None 2024-02-10T06:43:34.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:33 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/3491394338' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T06:43:34.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:33 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/3491394338' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T06:43:34.464 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953497 got 137438953500 for osd.4 2024-02-10T06:43:34.464 DEBUG:teuthology.parallel:result is None 2024-02-10T06:43:34.862 INFO:teuthology.orchestra.run.smithi008.stdout:60129542189 2024-02-10T06:43:35.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:34 smithi008 ceph-mon[24646]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:35.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:34 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/3120384840' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T06:43:35.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:34 smithi029 ceph-mon[27077]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:35.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:34 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/3120384840' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T06:43:35.881 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542186 got 60129542189 for osd.1 2024-02-10T06:43:35.881 DEBUG:teuthology.parallel:result is None 2024-02-10T06:43:36.940 INFO:teuthology.orchestra.run.smithi008.stdout:38654705716 2024-02-10T06:43:37.255 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:36 smithi008 ceph-mon[24646]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:37.255 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:36 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/3380124666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T06:43:37.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:36 smithi029 ceph-mon[27077]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:37.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:36 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/3380124666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T06:43:37.567 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705714 got 38654705716 for osd.0 2024-02-10T06:43:37.567 DEBUG:teuthology.parallel:result is None 2024-02-10T06:43:37.567 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-10T06:43:37.567 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T06:43:37.792 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:39.193 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:38 smithi008 ceph-mon[24646]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:39.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:38 smithi029 ceph-mon[27077]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:39.602 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:43:39.604 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T06:43:40.227 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:39 smithi008 ceph-mon[24646]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:43:40.289 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-02-10T06:43:37.909751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48864,"kb_used_data":6368,"kb_used_omap":0,"kb_used_meta":42432,"kb_avail":749912352,"statfs":{"total":767960285184,"available":767910248448,"internally_reserved":0,"allocated":6520832,"data_stored":3570848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43450368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002393"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":226,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-02-10T06:42:44.810372+0000","last_change":"2024-02-10T06:42:44.810372+0000","last_active":"2024-02-10T06:42:44.810372+0000","last_peered":"2024-02-10T06:42:44.810372+0000","last_clean":"2024-02-10T06:42:44.810372+0000","last_became_active":"2024-02-10T06:42:44.810016+0000","last_became_peered":"2024-02-10T06:42:44.810016+0000","last_unstale":"2024-02-10T06:42:44.810372+0000","last_undegraded":"2024-02-10T06:42:44.810372+0000","last_fullsized":"2024-02-10T06:42:44.810372+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T06:40:32.239517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T06:40:32.239517+0000","last_clean_scrub_stamp":"2024-02-10T06:40:32.239517+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:43:46.984624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6060,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739092,"statfs":{"total":95995035648,"available":95988830208,"internally_reserved":0,"allocated":1150976,"data_stored":777142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]}]},{"osd":6,"up_from":48,"seq":206158430228,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6184,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738968,"statfs":{"total":95995035648,"available":95988703232,"internally_reserved":0,"allocated":1146880,"data_stored":776877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.52400000000000002}]},{"osd":1,"last update":"Sat Feb 10 06:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.56399999999999995}]},{"osd":2,"last update":"Sat Feb 10 06:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.63100000000000001}]},{"osd":3,"last update":"Sat Feb 10 06:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.59799999999999998}]},{"osd":4,"last update":"Sat Feb 10 06:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.435},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.497}]},{"osd":5,"last update":"Sat Feb 10 06:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.46300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6372,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:43:07 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.36299999999999999,"15min":0.33400000000000002},"min":{"1min":0.23499999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.371,"15min":0.34499999999999997},"min":{"1min":0.25800000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.42399999999999999}]},{"osd":2,"last update":"Sat Feb 10 06:43:37 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40600000000000003,"15min":0.38800000000000001},"min":{"1min":0.27100000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.36699999999999999},"min":{"1min":0.28699999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.442}]},{"osd":3,"last update":"Sat Feb 10 06:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42299999999999999,"15min":0.42099999999999999},"min":{"1min":0.33100000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46700000000000003,"15min":0.46899999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.59999999999999998,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.46200000000000002}]},{"osd":4,"last update":"Sat Feb 10 06:43:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63200000000000001,"15min":0.626},"min":{"1min":0.38100000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.624},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.747,"15min":0.753},"min":{"1min":0.44500000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2270000000000001,"5min":3.5720000000000001,"15min":3.5720000000000001},"last":0.65400000000000003}]},{"osd":5,"last update":"Sat Feb 10 06:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.57899999999999996}]},{"osd":6,"last update":"Sat Feb 10 06:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.59199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6956,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738196,"statfs":{"total":95995035648,"available":95987912704,"internally_reserved":0,"allocated":1150976,"data_stored":777142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 06:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.33900000000000002,"15min":0.317},"min":{"1min":0.22700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51900000000000002,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.375,"15min":0.34399999999999997},"min":{"1min":0.28899999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Feb 10 06:42:33 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.38,"15min":0.36899999999999999},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.504},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.36099999999999999,"15min":0.35299999999999998},"min":{"1min":0.26400000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.48299999999999998}]},{"osd":3,"last update":"Sat Feb 10 06:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41499999999999998,"15min":0.41699999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.56100000000000005}]},{"osd":4,"last update":"Sat Feb 10 06:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.46899999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76700000000000002,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.623,"5min":1.0269999999999999,"15min":1.0940000000000001},"min":{"1min":0.47299999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.90200000000000002,"5min":8.6530000000000005,"15min":8.6530000000000005},"last":0.60299999999999998}]},{"osd":5,"last update":"Sat Feb 10 06:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.75900000000000001}]},{"osd":6,"last update":"Sat Feb 10 06:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.64000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]}]},{"osd":2,"up_from":19,"seq":81604378664,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6240,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738912,"statfs":{"total":95995035648,"available":95988645888,"internally_reserved":0,"allocated":614400,"data_stored":247949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.34799999999999998,"15min":0.32800000000000001},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.48699999999999999,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.378,"15min":0.35999999999999999},"min":{"1min":0.26100000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.44700000000000001}]},{"osd":1,"last update":"Sat Feb 10 06:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.39600000000000002,"15min":0.37},"min":{"1min":0.27500000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.36499999999999999},"min":{"1min":0.246,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.42099999999999999}]},{"osd":3,"last update":"Sat Feb 10 06:43:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.442,"15min":0.44},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.45400000000000001,"15min":0.45200000000000001},"min":{"1min":0.315,"5min":0.314,"15min":0.314},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.47799999999999998}]},{"osd":4,"last update":"Sat Feb 10 06:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.63100000000000001,"15min":0.61899999999999999},"min":{"1min":0.38100000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.623},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.64000000000000001,"15min":0.63300000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.65500000000000003}]},{"osd":5,"last update":"Sat Feb 10 06:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.69599999999999995}]},{"osd":6,"last update":"Sat Feb 10 06:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117026,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5664,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739488,"statfs":{"total":95995035648,"available":95989235712,"internally_reserved":0,"allocated":614400,"data_stored":247949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:43:03 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.42899999999999999,"15min":0.42599999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.379},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.433,"15min":0.43099999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Feb 10 06:43:03 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.44400000000000001,"15min":0.439},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.316},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.437,"15min":0.42899999999999999},"min":{"1min":0.33100000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.42899999999999999}]},{"osd":2,"last update":"Sat Feb 10 06:43:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.47899999999999998,"15min":0.47299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.45500000000000002,"15min":0.44900000000000001},"min":{"1min":0.32500000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Sat Feb 10 06:43:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.503},{"interface":"front","average":{"1min":0.63,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.38700000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.63500000000000001}]},{"osd":5,"last update":"Sat Feb 10 06:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.73399999999999999}]},{"osd":6,"last update":"Sat Feb 10 06:43:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.76400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]}]},{"osd":4,"up_from":32,"seq":137438953502,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5728,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739424,"statfs":{"total":95995035648,"available":95989170176,"internally_reserved":0,"allocated":614400,"data_stored":247949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:43:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54400000000000004,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.72799999999999998,"5min":0.754,"15min":0.754},"last":0.60499999999999998}]},{"osd":1,"last update":"Sat Feb 10 06:43:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58199999999999996,"15min":0.58399999999999996},"min":{"1min":0.41699999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.75900000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58999999999999997,"15min":0.59299999999999997},"min":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.62}]},{"osd":2,"last update":"Sat Feb 10 06:43:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69999999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.499},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55800000000000005,"15min":0.56100000000000005},"min":{"1min":0.38700000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.69299999999999995,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.55600000000000005}]},{"osd":3,"last update":"Sat Feb 10 06:43:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57299999999999995,"15min":0.57899999999999996},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68200000000000005,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55400000000000005,"15min":0.55500000000000005},"min":{"1min":0.42399999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67800000000000005,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.51300000000000001}]},{"osd":5,"last update":"Sat Feb 10 06:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.36799999999999999}]},{"osd":6,"last update":"Sat Feb 10 06:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.33300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]},{"osd":5,"up_from":40,"seq":171798691865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5660,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739492,"statfs":{"total":95995035648,"available":95989239808,"internally_reserved":0,"allocated":610304,"data_stored":247626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.33800000000000002}]},{"osd":1,"last update":"Sat Feb 10 06:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.57399999999999995}]},{"osd":2,"last update":"Sat Feb 10 06:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.46000000000000002}]},{"osd":3,"last update":"Sat Feb 10 06:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.61699999999999999}]},{"osd":4,"last update":"Sat Feb 10 06:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.496}]},{"osd":6,"last update":"Sat Feb 10 06:43:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T06:43:40.290 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T06:43:40.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:39 smithi029 ceph-mon[27077]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:43:40.519 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:41.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:40 smithi008 ceph-mon[24646]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:41.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:40 smithi029 ceph-mon[27077]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:42.270 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:43:42.271 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T06:43:42.927 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-02-10T06:43:41.910695+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48864,"kb_used_data":6368,"kb_used_omap":0,"kb_used_meta":42432,"kb_avail":749912352,"statfs":{"total":767960285184,"available":767910248448,"internally_reserved":0,"allocated":6520832,"data_stored":3570848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43450368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002515"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":226,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-02-10T06:42:44.810372+0000","last_change":"2024-02-10T06:42:44.810372+0000","last_active":"2024-02-10T06:42:44.810372+0000","last_peered":"2024-02-10T06:42:44.810372+0000","last_clean":"2024-02-10T06:42:44.810372+0000","last_became_active":"2024-02-10T06:42:44.810016+0000","last_became_peered":"2024-02-10T06:42:44.810016+0000","last_unstale":"2024-02-10T06:42:44.810372+0000","last_undegraded":"2024-02-10T06:42:44.810372+0000","last_fullsized":"2024-02-10T06:42:44.810372+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T06:40:32.239517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T06:40:32.239517+0000","last_clean_scrub_stamp":"2024-02-10T06:40:32.239517+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:43:46.984624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168591,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6060,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739092,"statfs":{"total":95995035648,"available":95988830208,"internally_reserved":0,"allocated":1150976,"data_stored":777142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]}]},{"osd":6,"up_from":48,"seq":206158430229,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6184,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738968,"statfs":{"total":95995035648,"available":95988703232,"internally_reserved":0,"allocated":1146880,"data_stored":776877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.42199999999999999}]},{"osd":1,"last update":"Sat Feb 10 06:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.65800000000000003}]},{"osd":2,"last update":"Sat Feb 10 06:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.55000000000000004}]},{"osd":3,"last update":"Sat Feb 10 06:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.70299999999999996}]},{"osd":4,"last update":"Sat Feb 10 06:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.72499999999999998}]},{"osd":5,"last update":"Sat Feb 10 06:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.628},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.68300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6372,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:43:07 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.36299999999999999,"15min":0.33400000000000002},"min":{"1min":0.23499999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.371,"15min":0.34499999999999997},"min":{"1min":0.25800000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.42399999999999999}]},{"osd":2,"last update":"Sat Feb 10 06:43:37 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40600000000000003,"15min":0.38800000000000001},"min":{"1min":0.27100000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.36699999999999999},"min":{"1min":0.28699999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.442}]},{"osd":3,"last update":"Sat Feb 10 06:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42299999999999999,"15min":0.42099999999999999},"min":{"1min":0.33100000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46700000000000003,"15min":0.46899999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.59999999999999998,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.46200000000000002}]},{"osd":4,"last update":"Sat Feb 10 06:43:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63200000000000001,"15min":0.626},"min":{"1min":0.38100000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.624},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.747,"15min":0.753},"min":{"1min":0.44500000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2270000000000001,"5min":3.5720000000000001,"15min":3.5720000000000001},"last":0.65400000000000003}]},{"osd":5,"last update":"Sat Feb 10 06:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.57899999999999996}]},{"osd":6,"last update":"Sat Feb 10 06:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.59199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705717,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6956,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738196,"statfs":{"total":95995035648,"available":95987912704,"internally_reserved":0,"allocated":1150976,"data_stored":777142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 06:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.33900000000000002,"15min":0.317},"min":{"1min":0.22700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51900000000000002,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.438,"5min":0.375,"15min":0.34399999999999997},"min":{"1min":0.28899999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Feb 10 06:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38300000000000001,"15min":0.37},"min":{"1min":0.25800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.56799999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.372},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.36799999999999999,"15min":0.35599999999999998},"min":{"1min":0.251,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.39400000000000002}]},{"osd":3,"last update":"Sat Feb 10 06:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41499999999999998,"15min":0.41699999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.47199999999999998}]},{"osd":4,"last update":"Sat Feb 10 06:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.46899999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76700000000000002,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.623,"5min":1.0269999999999999,"15min":1.0940000000000001},"min":{"1min":0.47299999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.90200000000000002,"5min":8.6530000000000005,"15min":8.6530000000000005},"last":0.63700000000000001}]},{"osd":5,"last update":"Sat Feb 10 06:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.58099999999999996}]},{"osd":6,"last update":"Sat Feb 10 06:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.623},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.59699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":2,"up_from":19,"seq":81604378665,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6240,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738912,"statfs":{"total":95995035648,"available":95988645888,"internally_reserved":0,"allocated":614400,"data_stored":247949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.34799999999999998,"15min":0.32800000000000001},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.48699999999999999,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.378,"15min":0.35999999999999999},"min":{"1min":0.26100000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.44700000000000001}]},{"osd":1,"last update":"Sat Feb 10 06:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.39600000000000002,"15min":0.37},"min":{"1min":0.27500000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.36499999999999999},"min":{"1min":0.246,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.42099999999999999}]},{"osd":3,"last update":"Sat Feb 10 06:43:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.442,"15min":0.44},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.45400000000000001,"15min":0.45200000000000001},"min":{"1min":0.315,"5min":0.314,"15min":0.314},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.47799999999999998}]},{"osd":4,"last update":"Sat Feb 10 06:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.63100000000000001,"15min":0.61899999999999999},"min":{"1min":0.38100000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.623},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.64000000000000001,"15min":0.63300000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.65500000000000003}]},{"osd":5,"last update":"Sat Feb 10 06:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.69599999999999995}]},{"osd":6,"last update":"Sat Feb 10 06:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5664,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739488,"statfs":{"total":95995035648,"available":95989235712,"internally_reserved":0,"allocated":614400,"data_stored":247949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:43:03 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.42899999999999999,"15min":0.42599999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.307},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.433,"15min":0.43099999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.34200000000000003}]},{"osd":1,"last update":"Sat Feb 10 06:43:03 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.44400000000000001,"15min":0.439},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.437,"15min":0.42899999999999999},"min":{"1min":0.33100000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.27400000000000002}]},{"osd":2,"last update":"Sat Feb 10 06:43:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.47899999999999998,"15min":0.47299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.314},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.45500000000000002,"15min":0.44900000000000001},"min":{"1min":0.32500000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.44900000000000001}]},{"osd":4,"last update":"Sat Feb 10 06:43:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.38700000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.56499999999999995}]},{"osd":5,"last update":"Sat Feb 10 06:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.63100000000000001}]},{"osd":6,"last update":"Sat Feb 10 06:43:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]}]},{"osd":4,"up_from":32,"seq":137438953502,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5728,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739424,"statfs":{"total":95995035648,"available":95989170176,"internally_reserved":0,"allocated":614400,"data_stored":247949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:43:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54400000000000004,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.72799999999999998,"5min":0.754,"15min":0.754},"last":0.60499999999999998}]},{"osd":1,"last update":"Sat Feb 10 06:43:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58199999999999996,"15min":0.58399999999999996},"min":{"1min":0.41699999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.75900000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58999999999999997,"15min":0.59299999999999997},"min":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.62}]},{"osd":2,"last update":"Sat Feb 10 06:43:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69999999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.499},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55800000000000005,"15min":0.56100000000000005},"min":{"1min":0.38700000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.69299999999999995,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.55600000000000005}]},{"osd":3,"last update":"Sat Feb 10 06:43:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57299999999999995,"15min":0.57899999999999996},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68200000000000005,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55400000000000005,"15min":0.55500000000000005},"min":{"1min":0.42399999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67800000000000005,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.51300000000000001}]},{"osd":5,"last update":"Sat Feb 10 06:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.36799999999999999}]},{"osd":6,"last update":"Sat Feb 10 06:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.33300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]},{"osd":5,"up_from":40,"seq":171798691866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5660,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739492,"statfs":{"total":95995035648,"available":95989239808,"internally_reserved":0,"allocated":610304,"data_stored":247626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.56100000000000005}]},{"osd":1,"last update":"Sat Feb 10 06:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.53200000000000003}]},{"osd":2,"last update":"Sat Feb 10 06:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.57399999999999995}]},{"osd":3,"last update":"Sat Feb 10 06:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.54300000000000004}]},{"osd":4,"last update":"Sat Feb 10 06:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.47799999999999998}]},{"osd":6,"last update":"Sat Feb 10 06:43:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.41099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T06:43:42.929 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-10T06:43:42.929 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-10T06:43:42.929 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-10T06:43:42.929 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph health --format=json 2024-02-10T06:43:43.160 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/mon.smithi008/config 2024-02-10T06:43:43.282 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:42 smithi008 ceph-mon[24646]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:43.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:42 smithi008 ceph-mon[24646]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:43:43.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:42 smithi029 ceph-mon[27077]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:43.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:42 smithi029 ceph-mon[27077]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:43:44.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:43 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:43:44.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:43 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:43:44.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:43 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:43:44.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:43 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:43:44.257 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:43 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:43:44.257 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:43 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:43:44.257 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:43 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:43:44.257 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:43 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:43:44.970 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:43:44.970 INFO:teuthology.orchestra.run.smithi008.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-10T06:43:45.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:44 smithi008 ceph-mon[24646]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:45.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:44 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/4182829198' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T06:43:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:44 smithi029 ceph-mon[27077]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:44 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/4182829198' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T06:43:45.564 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-10T06:43:45.565 INFO:tasks.cephadm:Setup complete, yielding 2024-02-10T06:43:45.565 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:43:45.574 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T06:43:45.575 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph orch status' 2024-02-10T06:43:47.257 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:46 smithi008 ceph-mon[24646]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:47.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:46 smithi029 ceph-mon[27077]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:47.565 INFO:teuthology.orchestra.run.smithi008.stdout:Backend: cephadm 2024-02-10T06:43:47.565 INFO:teuthology.orchestra.run.smithi008.stdout:Available: Yes 2024-02-10T06:43:47.565 INFO:teuthology.orchestra.run.smithi008.stdout:Paused: No 2024-02-10T06:43:48.153 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:47 smithi008 ceph-mon[24646]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:43:48.179 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-10T06:43:48.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:47 smithi029 ceph-mon[27077]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:43:49.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:48 smithi008 ceph-mon[24646]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:49.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:48 smithi029 ceph-mon[27077]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:50.072 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:43:50.072 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (4m) 2m ago 5m 17.6M - ba2b418f427c d582fd72ee55 2024-02-10T06:43:50.072 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (5m) 2m ago 5m 6405k - 17.2.0 e1d6a67b021e 5fa2e5bdd7cc 2024-02-10T06:43:50.072 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (5m) 69s ago 5m 6405k - 17.2.0 e1d6a67b021e 7527ea77b5cc 2024-02-10T06:43:50.072 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (4m) 2m ago 5m 46.4M - 8.3.5 dad864ee21e9 c9887c4d30ec 2024-02-10T06:43:50.072 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:9283 running (7m) 2m ago 7m 448M - 17.2.0 e1d6a67b021e b0d0b31e96f4 2024-02-10T06:43:50.072 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283 running (5m) 69s ago 5m 413M - 17.2.0 e1d6a67b021e 4847b9108037 2024-02-10T06:43:50.072 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (7m) 2m ago 7m 41.7M 2048M 17.2.0 e1d6a67b021e 061312907e4b 2024-02-10T06:43:50.072 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (5m) 69s ago 5m 42.0M 2048M 17.2.0 e1d6a67b021e a790a61584bf 2024-02-10T06:43:50.072 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (5m) 2m ago 5m 16.6M - 1dbe0e931976 3c06441739f8 2024-02-10T06:43:50.072 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (4m) 69s ago 4m 17.2M - 1dbe0e931976 6e5039760c8a 2024-02-10T06:43:50.073 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (4m) 2m ago 4m 36.0M 3217M 17.2.0 e1d6a67b021e 139b7a92bfc4 2024-02-10T06:43:50.073 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (3m) 2m ago 3m 35.4M 3217M 17.2.0 e1d6a67b021e 0f1119e17a52 2024-02-10T06:43:50.073 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (3m) 2m ago 3m 34.7M 3217M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:43:50.073 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (2m) 2m ago 2m 25.9M 3217M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:43:50.073 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (2m) 69s ago 2m 36.0M 3985M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:43:50.073 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (2m) 69s ago 2m 35.4M 3985M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:43:50.073 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (105s) 69s ago 104s 30.3M 3985M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:43:50.073 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (75s) 69s ago 74s 10.1M 3985M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:43:50.073 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (4m) 2m ago 4m 58.7M - 514e6a882f6e eedfd13fcab7 2024-02-10T06:43:50.714 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-10T06:43:51.252 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:50 smithi008 ceph-mon[24646]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:51.252 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:50 smithi008 ceph-mon[24646]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:43:51.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:50 smithi029 ceph-mon[27077]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:51.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:50 smithi029 ceph-mon[27077]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:43:52.646 INFO:teuthology.orchestra.run.smithi008.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-10T06:43:52.646 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-02-10T06:43:52.646 INFO:teuthology.orchestra.run.smithi008.stdout:crash 2/2 2m ago 6m * 2024-02-10T06:43:52.646 INFO:teuthology.orchestra.run.smithi008.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-02-10T06:43:52.646 INFO:teuthology.orchestra.run.smithi008.stdout:mgr 2/2 2m ago 6m count:2 2024-02-10T06:43:52.646 INFO:teuthology.orchestra.run.smithi008.stdout:mon 2/2 2m ago 5m smithi008:172.21.15.8=smithi008;smithi029:172.21.15.29=smithi029;count:2 2024-02-10T06:43:52.646 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-02-10T06:43:52.646 INFO:teuthology.orchestra.run.smithi008.stdout:osd 8 2m ago - 2024-02-10T06:43:52.646 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-02-10T06:43:53.254 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:53 smithi008 ceph-mon[24646]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:53.255 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:53 smithi008 ceph-mon[24646]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:43:53.279 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-10T06:43:53.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:53 smithi029 ceph-mon[27077]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:53.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:53 smithi029 ceph-mon[27077]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:43:54.921 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:54 smithi008 ceph-mon[24646]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:54 smithi029 ceph-mon[27077]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:55.349 INFO:teuthology.orchestra.run.smithi008.stdout:HOST ADDR LABELS STATUS 2024-02-10T06:43:55.349 INFO:teuthology.orchestra.run.smithi008.stdout:smithi008 172.21.15.8 2024-02-10T06:43:55.350 INFO:teuthology.orchestra.run.smithi008.stdout:smithi029 172.21.15.29 2024-02-10T06:43:55.350 INFO:teuthology.orchestra.run.smithi008.stdout:2 hosts in cluster 2024-02-10T06:43:56.040 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-10T06:43:56.068 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:55 smithi008 ceph-mon[24646]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:43:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:55 smithi029 ceph-mon[27077]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:43:57.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:56 smithi008 ceph-mon[24646]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:57.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:56 smithi029 ceph-mon[27077]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:58.006 INFO:teuthology.orchestra.run.smithi008.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-02-10T06:43:58.006 INFO:teuthology.orchestra.run.smithi008.stdout:smithi008 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476004A480DGN 480G LVM detected, locked 2024-02-10T06:43:58.006 INFO:teuthology.orchestra.run.smithi008.stdout:smithi029 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7512009B480DGN 480G LVM detected, locked 2024-02-10T06:43:58.594 INFO:teuthology.run_tasks:Running task vip.exec... 2024-02-10T06:43:58.602 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T06:43:58.602 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-02-10T06:43:58.664 INFO:teuthology.orchestra.run.smithi008.stderr:+ systemctl stop nfs-server 2024-02-10T06:43:58.672 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi029.front.sepia.ceph.com 2024-02-10T06:43:58.672 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-02-10T06:43:58.703 INFO:teuthology.orchestra.run.smithi029.stderr:+ systemctl stop nfs-server 2024-02-10T06:43:58.711 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:43:58.720 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T06:43:58.721 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph fs volume create foofs' 2024-02-10T06:43:59.279 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:58 smithi008 ceph-mon[24646]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:59.280 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:43:58 smithi008 ceph-mon[24646]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:43:59.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:58 smithi029 ceph-mon[27077]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:43:59.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:43:58 smithi029 ceph-mon[27077]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:01.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:01 smithi029 ceph-mon[27077]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:44:01.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:01 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-02-10T06:44:01.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:01 smithi008 ceph-mon[24646]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:44:01.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:01 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-02-10T06:44:02.436 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:02 smithi029 ceph-mon[27077]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:02.436 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:02 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-02-10T06:44:02.436 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:02 smithi029 ceph-mon[27077]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T06:44:02.436 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:02 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-02-10T06:44:02.499 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:02 smithi008 ceph-mon[24646]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:02.499 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:02 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-02-10T06:44:02.499 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:02 smithi008 ceph-mon[24646]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T06:44:02.499 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:02 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-02-10T06:44:02.499 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:02 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008[24642]: 2024-02-10T06:44:02.110+0000 7f974e46a700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T06:44:02.897 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-02-10T06:44:02.906 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-02-10T06:44:02.907 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch ls -f json 2024-02-10T06:44:03.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: pgmap v218: 33 pgs: 16 creating+peering, 16 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:44:03.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-02-10T06:44:03.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T06:44:03.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-02-10T06:44:03.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T06:44:03.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-10T06:44:03.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T06:44:03.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-02-10T06:44:03.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: fsmap foofs:0 2024-02-10T06:44:03.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: Saving service mds.foofs spec with placement count:2 2024-02-10T06:44:03.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:03.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:44:03.146 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:03.146 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:44:03.146 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:03.146 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:03.146 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.vfwcte", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:44:03.146 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.vfwcte", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:44:03.146 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:03.146 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:03 smithi008 ceph-mon[24646]: Deploying daemon mds.foofs.smithi008.vfwcte on smithi008 2024-02-10T06:44:03.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: pgmap v218: 33 pgs: 16 creating+peering, 16 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:44:03.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-02-10T06:44:03.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T06:44:03.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-02-10T06:44:03.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T06:44:03.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-10T06:44:03.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T06:44:03.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-02-10T06:44:03.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: fsmap foofs:0 2024-02-10T06:44:03.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: Saving service mds.foofs spec with placement count:2 2024-02-10T06:44:03.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:03.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:44:03.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:03.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:44:03.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:03.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:03.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.vfwcte", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:44:03.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.vfwcte", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:44:03.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:03.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:03 smithi029 ceph-mon[27077]: Deploying daemon mds.foofs.smithi008.vfwcte on smithi008 2024-02-10T06:44:04.452 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:04 smithi008 ceph-mon[24646]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T06:44:04.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:04 smithi029 ceph-mon[27077]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T06:44:05.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:05 smithi029 ceph-mon[27077]: pgmap v222: 65 pgs: 20 creating+peering, 21 unknown, 24 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:44:05.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:05 smithi029 ceph-mon[27077]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T06:44:05.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:05 smithi008 ceph-mon[24646]: pgmap v222: 65 pgs: 20 creating+peering, 21 unknown, 24 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:44:05.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:05 smithi008 ceph-mon[24646]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T06:44:06.357 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:44:06.358 INFO:teuthology.orchestra.run.smithi008.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-02-10T06:37:19.963271Z", "last_refresh": "2024-02-10T06:41:06.947522Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-02-10T06:37:13.131505Z", "last_refresh": "2024-02-10T06:41:06.947591Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-02-10T06:37:16.569177Z", "last_refresh": "2024-02-10T06:41:06.947657Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-02-10T06:44:02.127407Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-02-10T06:44:02.121603Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-02-10T06:37:11.800528Z", "last_refresh": "2024-02-10T06:41:06.947445Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi008:172.21.15.8=smithi008", "smithi029:172.21.15.29=smithi029"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-02-10T06:38:03.107535Z", "last_refresh": "2024-02-10T06:41:06.947295Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-02-10T06:37:18.394994Z", "last_refresh": "2024-02-10T06:41:06.947722Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-02-10T06:41:06.947849Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-02-10T06:37:14.787812Z", "last_refresh": "2024-02-10T06:41:06.947785Z", "ports": [9095], "running": 1, "size": 1}}] 2024-02-10T06:44:07.039 INFO:tasks.cephadm:mds.foofs has 0/2 2024-02-10T06:44:07.401 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:07 smithi029 ceph-mon[27077]: pgmap v224: 65 pgs: 20 creating+peering, 21 unknown, 24 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:44:07.401 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:07 smithi029 ceph-mon[27077]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:44:07.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:07 smithi008 ceph-mon[24646]: pgmap v224: 65 pgs: 20 creating+peering, 21 unknown, 24 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:44:07.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:07 smithi008 ceph-mon[24646]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:44:08.040 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch ls -f json 2024-02-10T06:44:08.364 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:08 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:08.364 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:08 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi029.bwmcyt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:44:08.364 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:08 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi029.bwmcyt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:44:08.364 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:08 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:08.365 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:08 smithi008 ceph-mon[24646]: Deploying daemon mds.foofs.smithi029.bwmcyt on smithi029 2024-02-10T06:44:08.365 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:08 smithi008 ceph-mon[24646]: mds.? [v2:172.21.15.8:6834/1608640408,v1:172.21.15.8:6835/1608640408] up:boot 2024-02-10T06:44:08.365 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:08 smithi008 ceph-mon[24646]: daemon mds.foofs.smithi008.vfwcte assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-02-10T06:44:08.365 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:08 smithi008 ceph-mon[24646]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-10T06:44:08.365 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:08 smithi008 ceph-mon[24646]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-10T06:44:08.365 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:08 smithi008 ceph-mon[24646]: Cluster is now healthy 2024-02-10T06:44:08.365 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:08 smithi008 ceph-mon[24646]: fsmap foofs:0 1 up:standby 2024-02-10T06:44:08.365 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:08 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.vfwcte"}]: dispatch 2024-02-10T06:44:08.365 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:08 smithi008 ceph-mon[24646]: fsmap foofs:1 {0=foofs.smithi008.vfwcte=up:creating} 2024-02-10T06:44:08.365 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:08 smithi008 ceph-mon[24646]: daemon mds.foofs.smithi008.vfwcte is now active in filesystem foofs as rank 0 2024-02-10T06:44:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:08 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:08 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi029.bwmcyt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:44:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:08 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi029.bwmcyt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:44:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:08 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:08 smithi029 ceph-mon[27077]: Deploying daemon mds.foofs.smithi029.bwmcyt on smithi029 2024-02-10T06:44:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:08 smithi029 ceph-mon[27077]: mds.? [v2:172.21.15.8:6834/1608640408,v1:172.21.15.8:6835/1608640408] up:boot 2024-02-10T06:44:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:08 smithi029 ceph-mon[27077]: daemon mds.foofs.smithi008.vfwcte assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-02-10T06:44:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:08 smithi029 ceph-mon[27077]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-10T06:44:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:08 smithi029 ceph-mon[27077]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-10T06:44:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:08 smithi029 ceph-mon[27077]: Cluster is now healthy 2024-02-10T06:44:08.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:08 smithi029 ceph-mon[27077]: fsmap foofs:0 1 up:standby 2024-02-10T06:44:08.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:08 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.vfwcte"}]: dispatch 2024-02-10T06:44:08.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:08 smithi029 ceph-mon[27077]: fsmap foofs:1 {0=foofs.smithi008.vfwcte=up:creating} 2024-02-10T06:44:08.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:08 smithi029 ceph-mon[27077]: daemon mds.foofs.smithi008.vfwcte is now active in filesystem foofs as rank 0 2024-02-10T06:44:09.589 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:09 smithi008 ceph-mon[24646]: pgmap v225: 65 pgs: 65 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:44:09.589 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:09 smithi008 ceph-mon[24646]: mds.? [v2:172.21.15.8:6834/1608640408,v1:172.21.15.8:6835/1608640408] up:active 2024-02-10T06:44:09.589 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:09 smithi008 ceph-mon[24646]: fsmap foofs:1 {0=foofs.smithi008.vfwcte=up:active} 2024-02-10T06:44:09.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:09 smithi029 ceph-mon[27077]: pgmap v225: 65 pgs: 65 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:44:09.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:09 smithi029 ceph-mon[27077]: mds.? [v2:172.21.15.8:6834/1608640408,v1:172.21.15.8:6835/1608640408] up:active 2024-02-10T06:44:09.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:09 smithi029 ceph-mon[27077]: fsmap foofs:1 {0=foofs.smithi008.vfwcte=up:active} 2024-02-10T06:44:10.180 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:44:10.180 INFO:teuthology.orchestra.run.smithi008.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-02-10T06:37:19.963271Z", "last_refresh": "2024-02-10T06:41:06.947522Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-02-10T06:37:13.131505Z", "last_refresh": "2024-02-10T06:41:06.947591Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-02-10T06:37:16.569177Z", "last_refresh": "2024-02-10T06:41:06.947657Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-02-10T06:44:02.127407Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-02-10T06:44:02.121603Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-02-10T06:37:11.800528Z", "last_refresh": "2024-02-10T06:41:06.947445Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi008:172.21.15.8=smithi008", "smithi029:172.21.15.29=smithi029"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-02-10T06:38:03.107535Z", "last_refresh": "2024-02-10T06:41:06.947295Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-02-10T06:37:18.394994Z", "last_refresh": "2024-02-10T06:41:06.947722Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-02-10T06:41:06.947849Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-02-10T06:37:14.787812Z", "last_refresh": "2024-02-10T06:41:06.947785Z", "ports": [9095], "running": 1, "size": 1}}] 2024-02-10T06:44:10.775 INFO:tasks.cephadm:mds.foofs has 0/2 2024-02-10T06:44:11.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:11 smithi008 ceph-mon[24646]: pgmap v226: 65 pgs: 65 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:44:11.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:11 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:11.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:11 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:44:11.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:11 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:11.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:11 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:44:11.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:11 smithi008 ceph-mon[24646]: from='client.14506 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:44:11.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:11 smithi008 ceph-mon[24646]: mds.? [v2:172.21.15.29:6832/1338152565,v1:172.21.15.29:6833/1338152565] up:boot 2024-02-10T06:44:11.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:11 smithi008 ceph-mon[24646]: fsmap foofs:1 {0=foofs.smithi008.vfwcte=up:active} 1 up:standby 2024-02-10T06:44:11.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:11 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi029.bwmcyt"}]: dispatch 2024-02-10T06:44:11.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:11 smithi008 ceph-mon[24646]: fsmap foofs:1 {0=foofs.smithi008.vfwcte=up:active} 1 up:standby 2024-02-10T06:44:11.100 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:11 smithi029 ceph-mon[27077]: pgmap v226: 65 pgs: 65 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:44:11.100 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:11 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:11.100 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:11 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:44:11.100 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:11 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:11.100 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:11 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:44:11.100 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:11 smithi029 ceph-mon[27077]: from='client.14506 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:44:11.100 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:11 smithi029 ceph-mon[27077]: mds.? [v2:172.21.15.29:6832/1338152565,v1:172.21.15.29:6833/1338152565] up:boot 2024-02-10T06:44:11.100 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:11 smithi029 ceph-mon[27077]: fsmap foofs:1 {0=foofs.smithi008.vfwcte=up:active} 1 up:standby 2024-02-10T06:44:11.101 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:11 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi029.bwmcyt"}]: dispatch 2024-02-10T06:44:11.101 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:11 smithi029 ceph-mon[27077]: fsmap foofs:1 {0=foofs.smithi008.vfwcte=up:active} 1 up:standby 2024-02-10T06:44:11.776 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch ls -f json 2024-02-10T06:44:13.444 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:13 smithi008 ceph-mon[24646]: pgmap v227: 65 pgs: 65 active+clean; 578 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 349 B/s wr, 3 op/s 2024-02-10T06:44:13.444 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:13 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:13.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:13 smithi029 ceph-mon[27077]: pgmap v227: 65 pgs: 65 active+clean; 578 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 349 B/s wr, 3 op/s 2024-02-10T06:44:13.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:13 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:14.470 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:44:14.470 INFO:teuthology.orchestra.run.smithi008.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-02-10T06:37:19.963271Z", "last_refresh": "2024-02-10T06:41:06.947522Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-02-10T06:37:13.131505Z", "last_refresh": "2024-02-10T06:41:06.947591Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-02-10T06:37:16.569177Z", "last_refresh": "2024-02-10T06:41:06.947657Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-02-10T06:44:02.127407Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-02-10T06:44:02.121603Z", "last_refresh": "2024-02-10T06:44:12.033224Z", "running": 1, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-02-10T06:37:11.800528Z", "last_refresh": "2024-02-10T06:41:06.947445Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi008:172.21.15.8=smithi008", "smithi029:172.21.15.29=smithi029"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-02-10T06:38:03.107535Z", "last_refresh": "2024-02-10T06:41:06.947295Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-02-10T06:37:18.394994Z", "last_refresh": "2024-02-10T06:41:06.947722Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-02-10T06:41:06.947849Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-02-10T06:37:14.787812Z", "last_refresh": "2024-02-10T06:41:06.947785Z", "ports": [9095], "running": 1, "size": 1}}] 2024-02-10T06:44:15.147 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:14 smithi008 ceph-mon[24646]: pgmap v228: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-02-10T06:44:15.147 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:14 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:15.147 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:14 smithi008 ceph-mon[24646]: from='client.14510 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:44:15.175 INFO:tasks.cephadm:mds.foofs has 1/2 2024-02-10T06:44:15.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:14 smithi029 ceph-mon[27077]: pgmap v228: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-02-10T06:44:15.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:14 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:15.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:14 smithi029 ceph-mon[27077]: from='client.14510 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:44:16.176 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch ls -f json 2024-02-10T06:44:17.149 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:16 smithi008 ceph-mon[24646]: pgmap v229: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T06:44:17.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:16 smithi029 ceph-mon[27077]: pgmap v229: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T06:44:18.597 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:44:18.598 INFO:teuthology.orchestra.run.smithi008.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-02-10T06:37:19.963271Z", "last_refresh": "2024-02-10T06:44:17.933346Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-02-10T06:37:13.131505Z", "last_refresh": "2024-02-10T06:44:12.032616Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-02-10T06:37:16.569177Z", "last_refresh": "2024-02-10T06:44:17.933515Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-02-10T06:44:02.127407Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-02-10T06:44:02.121603Z", "last_refresh": "2024-02-10T06:44:12.033224Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-02-10T06:37:11.800528Z", "last_refresh": "2024-02-10T06:44:12.032761Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi008:172.21.15.8=smithi008", "smithi029:172.21.15.29=smithi029"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-02-10T06:38:03.107535Z", "last_refresh": "2024-02-10T06:44:12.032837Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-02-10T06:37:18.394994Z", "last_refresh": "2024-02-10T06:44:12.032904Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-02-10T06:44:12.032969Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-02-10T06:37:14.787812Z", "last_refresh": "2024-02-10T06:44:17.933666Z", "ports": [9095], "running": 1, "size": 1}}] 2024-02-10T06:44:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:18 smithi029 ceph-mon[27077]: pgmap v230: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T06:44:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:18 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:18 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:18 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:44:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:18 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:18 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:44:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:18 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:18 smithi029 ceph-mon[27077]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:44:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:18 smithi008 ceph-mon[24646]: pgmap v230: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T06:44:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:18 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:18 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:18 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:44:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:18 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:18 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:44:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:18 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:18 smithi008 ceph-mon[24646]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:44:19.347 INFO:tasks.cephadm:mds.foofs has 2/2 2024-02-10T06:44:19.347 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:44:19.357 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T06:44:19.357 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-02-10T06:44:20.983 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:20 smithi008 ceph-mon[24646]: pgmap v231: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T06:44:21.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:20 smithi029 ceph-mon[27077]: pgmap v231: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T06:44:22.207 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:21 smithi008 ceph-mon[24646]: from='client.14518 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:22.207 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:21 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-02-10T06:44:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:21 smithi029 ceph-mon[27077]: from='client.14518 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:21 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-02-10T06:44:22.985 INFO:teuthology.orchestra.run.smithi008.stdout:NFS Cluster Created Successfully 2024-02-10T06:44:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:22 smithi029 ceph-mon[27077]: pgmap v232: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T06:44:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:22 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-02-10T06:44:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:22 smithi029 ceph-mon[27077]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T06:44:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:22 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-02-10T06:44:23.211 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:22 smithi008 ceph-mon[24646]: pgmap v232: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T06:44:23.211 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:22 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-02-10T06:44:23.212 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:22 smithi008 ceph-mon[24646]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T06:44:23.212 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:22 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-02-10T06:44:23.767 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-02-10T06:44:24.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-02-10T06:44:24.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T06:44:24.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: Saving service nfs.foo spec with placement count:2 2024-02-10T06:44:24.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:24.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:44:24.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:24.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:44:24.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:24.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:24.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: Creating key for client.nfs.foo.0.0.smithi008.pgujgu 2024-02-10T06:44:24.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.pgujgu", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-10T06:44:24.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.pgujgu", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-02-10T06:44:24.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-02-10T06:44:24.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-10T06:44:24.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-10T06:44:24.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:24.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-10T06:44:24.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-10T06:44:24.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: Rados config object exists: conf-nfs.foo 2024-02-10T06:44:24.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: Creating key for client.nfs.foo.0.0.smithi008.pgujgu-rgw 2024-02-10T06:44:24.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.pgujgu-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:44:24.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.pgujgu-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T06:44:24.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:24.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: Deploying daemon nfs.foo.0.0.smithi008.pgujgu on smithi008 2024-02-10T06:44:24.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:44:24.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:23 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:44:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-02-10T06:44:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T06:44:24.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: Saving service nfs.foo spec with placement count:2 2024-02-10T06:44:24.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:24.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:44:24.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:24.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:44:24.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:24.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:24.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: Creating key for client.nfs.foo.0.0.smithi008.pgujgu 2024-02-10T06:44:24.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.pgujgu", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-10T06:44:24.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.pgujgu", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-02-10T06:44:24.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-02-10T06:44:24.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-10T06:44:24.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-10T06:44:24.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:24.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-10T06:44:24.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-10T06:44:24.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: Rados config object exists: conf-nfs.foo 2024-02-10T06:44:24.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: Creating key for client.nfs.foo.0.0.smithi008.pgujgu-rgw 2024-02-10T06:44:24.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.pgujgu-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:44:24.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.pgujgu-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T06:44:24.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:24.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: Deploying daemon nfs.foo.0.0.smithi008.pgujgu on smithi008 2024-02-10T06:44:24.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:44:24.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:23 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:44:25.002 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:24 smithi008 ceph-mon[24646]: pgmap v235: 97 pgs: 3 creating+activating, 14 creating+peering, 6 unknown, 74 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 255 B/s wr, 0 op/s 2024-02-10T06:44:25.002 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:24 smithi008 ceph-mon[24646]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T06:44:25.002 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:24 smithi008 ceph-mon[24646]: mgrmap e20: smithi008.pxifbt(active, since 6m), standbys: smithi029.fuvrtf 2024-02-10T06:44:25.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:24 smithi029 ceph-mon[27077]: pgmap v235: 97 pgs: 3 creating+activating, 14 creating+peering, 6 unknown, 74 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 255 B/s wr, 0 op/s 2024-02-10T06:44:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:24 smithi029 ceph-mon[27077]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T06:44:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:24 smithi029 ceph-mon[27077]: mgrmap e20: smithi008.pxifbt(active, since 6m), standbys: smithi029.fuvrtf 2024-02-10T06:44:26.845 INFO:teuthology.orchestra.run.smithi008.stderr:Invalid command: Unexpected argument '--clusterid' 2024-02-10T06:44:26.845 INFO:teuthology.orchestra.run.smithi008.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-02-10T06:44:26.845 INFO:teuthology.orchestra.run.smithi008.stderr:Error EINVAL: invalid command 2024-02-10T06:44:27.094 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:26 smithi008 ceph-mon[24646]: pgmap v237: 97 pgs: 3 creating+activating, 14 creating+peering, 6 unknown, 74 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-02-10T06:44:27.208 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:44:27.208 INFO:teuthology.orchestra.run.smithi008.stdout: "bind": "/fake", 2024-02-10T06:44:27.208 INFO:teuthology.orchestra.run.smithi008.stdout: "fs": "foofs", 2024-02-10T06:44:27.208 INFO:teuthology.orchestra.run.smithi008.stdout: "path": "/", 2024-02-10T06:44:27.208 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster": "foo", 2024-02-10T06:44:27.208 INFO:teuthology.orchestra.run.smithi008.stdout: "mode": "RW" 2024-02-10T06:44:27.208 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:44:27.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:26 smithi029 ceph-mon[27077]: pgmap v237: 97 pgs: 3 creating+activating, 14 creating+peering, 6 unknown, 74 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-02-10T06:44:27.924 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-02-10T06:44:28.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:27 smithi008 ceph-mon[24646]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:28.239 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:27 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-02-10T06:44:28.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:27 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-02-10T06:44:28.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:27 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:28.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:27 smithi008 ceph-mon[24646]: Creating key for client.nfs.foo.1.0.smithi029.cvyqqw 2024-02-10T06:44:28.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:27 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi029.cvyqqw", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-10T06:44:28.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:27 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi029.cvyqqw", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-02-10T06:44:28.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:27 smithi008 ceph-mon[24646]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-02-10T06:44:28.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:27 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-10T06:44:28.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:27 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-10T06:44:28.240 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:27 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:28.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:27 smithi029 ceph-mon[27077]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:28.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:27 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-02-10T06:44:28.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:27 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-02-10T06:44:28.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:27 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:28.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:27 smithi029 ceph-mon[27077]: Creating key for client.nfs.foo.1.0.smithi029.cvyqqw 2024-02-10T06:44:28.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:27 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi029.cvyqqw", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-10T06:44:28.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:27 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi029.cvyqqw", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-02-10T06:44:28.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:27 smithi029 ceph-mon[27077]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-02-10T06:44:28.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:27 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-10T06:44:28.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:27 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-10T06:44:28.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:27 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:29.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:29 smithi008 ceph-mon[24646]: pgmap v238: 97 pgs: 3 creating+activating, 94 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-02-10T06:44:29.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:29 smithi029 ceph-mon[27077]: pgmap v238: 97 pgs: 3 creating+activating, 94 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-02-10T06:44:30.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:30 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-10T06:44:30.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:30 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-10T06:44:30.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:30 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi029.cvyqqw-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:44:30.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:30 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi029.cvyqqw-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T06:44:30.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:30 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:30.607 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:30 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-10T06:44:30.608 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:30 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-10T06:44:30.608 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:30 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi029.cvyqqw-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:44:30.608 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:30 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi029.cvyqqw-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T06:44:30.608 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:30 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:31.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:31 smithi008 ceph-mon[24646]: pgmap v239: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-02-10T06:44:31.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:31 smithi008 ceph-mon[24646]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:31.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:31 smithi008 ceph-mon[24646]: Rados config object exists: conf-nfs.foo 2024-02-10T06:44:31.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:31 smithi008 ceph-mon[24646]: Creating key for client.nfs.foo.1.0.smithi029.cvyqqw-rgw 2024-02-10T06:44:31.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:31 smithi008 ceph-mon[24646]: Deploying daemon nfs.foo.1.0.smithi029.cvyqqw on smithi029 2024-02-10T06:44:31.654 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:31 smithi029 ceph-mon[27077]: pgmap v239: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-02-10T06:44:31.654 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:31 smithi029 ceph-mon[27077]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:31.654 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:31 smithi029 ceph-mon[27077]: Rados config object exists: conf-nfs.foo 2024-02-10T06:44:31.654 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:31 smithi029 ceph-mon[27077]: Creating key for client.nfs.foo.1.0.smithi029.cvyqqw-rgw 2024-02-10T06:44:31.654 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:31 smithi029 ceph-mon[27077]: Deploying daemon nfs.foo.1.0.smithi029.cvyqqw on smithi029 2024-02-10T06:44:32.381 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:32 smithi029 ceph-mon[27077]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:32.525 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:32 smithi008 ceph-mon[24646]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:33 smithi029 ceph-mon[27077]: pgmap v240: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T06:44:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:33 smithi029 ceph-mon[27077]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:33 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:33 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:44:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:33 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:33 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:44:34.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:33 smithi008 ceph-mon[24646]: pgmap v240: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T06:44:34.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:33 smithi008 ceph-mon[24646]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:34.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:33 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:34.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:33 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:44:34.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:33 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:34.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:33 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:44:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:34 smithi029 ceph-mon[27077]: pgmap v241: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-02-10T06:44:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:34 smithi029 ceph-mon[27077]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:34 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:35.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:34 smithi008 ceph-mon[24646]: pgmap v241: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-02-10T06:44:35.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:34 smithi008 ceph-mon[24646]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:35.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:34 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:36.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:36 smithi008 ceph-mon[24646]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:36.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:36 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:36.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:36 smithi029 ceph-mon[27077]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:36.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:36 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:37.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:37 smithi029 ceph-mon[27077]: pgmap v242: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-02-10T06:44:37.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:37 smithi029 ceph-mon[27077]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:37.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:37 smithi008 ceph-mon[24646]: pgmap v242: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-02-10T06:44:37.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:37 smithi008 ceph-mon[24646]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:37.893 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo ?:2049 2/2 2s ago 14s count:2 2024-02-10T06:44:38.525 INFO:teuthology.run_tasks:Running task vip.exec... 2024-02-10T06:44:38.541 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T06:44:38.541 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-02-10T06:44:38.570 INFO:teuthology.orchestra.run.smithi008.stderr:+ mkdir /mnt/foo 2024-02-10T06:44:38.579 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-02-10T06:44:38.645 INFO:teuthology.orchestra.run.smithi008.stderr:++ hostname 2024-02-10T06:44:38.646 INFO:teuthology.orchestra.run.smithi008.stderr:+ mount -t nfs smithi008:/fake /mnt/foo -o sync 2024-02-10T06:44:38.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:38 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:38.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:38 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:38.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:38 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:38.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:38 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:44:38.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:38 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:38.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:38 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:44:38.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:38 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:38.782 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:38 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:39.063 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-02-10T06:44:39.130 INFO:teuthology.orchestra.run.smithi008.stderr:+ echo test 2024-02-10T06:44:39.140 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-02-10T06:44:39.206 INFO:teuthology.orchestra.run.smithi008.stderr:+ sync 2024-02-10T06:44:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:38 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:38 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:38 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:38 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:44:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:38 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:44:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:38 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:44:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:38 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:38 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:44:39.364 INFO:teuthology.run_tasks:Running task parallel... 2024-02-10T06:44:39.376 INFO:teuthology.task.parallel:starting parallel... 2024-02-10T06:44:39.377 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-10T06:44:39.377 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-10T06:44:39.378 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T06:44:39.378 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-10T06:44:39.378 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-10T06:44:39.378 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-02-10T06:44:39.379 INFO:teuthology.task.exec:Executing custom commands... 2024-02-10T06:44:39.379 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T06:44:39.379 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-02-10T06:44:39.501 INFO:teuthology.orchestra.run.smithi008.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-02-10T06:44:39.502 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:44:39.502 INFO:teuthology.orchestra.run.smithi008.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-02-10T06:44:39.502 INFO:teuthology.orchestra.run.smithi008.stdout:3 of 5 processes prepared for launch 0 sec 2024-02-10T06:44:39.602 INFO:teuthology.orchestra.run.smithi008.stdout:5 of 5 processes prepared for launch 0 sec 2024-02-10T06:44:39.602 INFO:teuthology.orchestra.run.smithi008.stdout:releasing clients 2024-02-10T06:44:39.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:39 smithi008 ceph-mon[24646]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:39.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:39 smithi008 ceph-mon[24646]: pgmap v243: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-02-10T06:44:40.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:39 smithi029 ceph-mon[27077]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:44:40.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:39 smithi029 ceph-mon[27077]: pgmap v243: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-02-10T06:44:40.602 INFO:teuthology.orchestra.run.smithi008.stdout: 5 302 55.47 MB/sec warmup 1 sec latency 30.459 ms 2024-02-10T06:44:41.602 INFO:teuthology.orchestra.run.smithi008.stdout: 5 625 52.07 MB/sec warmup 2 sec latency 23.124 ms 2024-02-10T06:44:41.905 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:41 smithi008 ceph-mon[24646]: pgmap v244: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-02-10T06:44:42.009 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:41 smithi029 ceph-mon[27077]: pgmap v244: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-02-10T06:44:42.602 INFO:teuthology.orchestra.run.smithi008.stdout: 5 1437 43.37 MB/sec warmup 3 sec latency 11.318 ms 2024-02-10T06:44:43.602 INFO:teuthology.orchestra.run.smithi008.stdout: 5 2525 35.96 MB/sec warmup 4 sec latency 8.889 ms 2024-02-10T06:44:43.609 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-10T06:44:43.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:43 smithi008 ceph-mon[24646]: pgmap v245: 97 pgs: 97 active+clean; 42 MiB data, 96 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 3.5 MiB/s wr, 192 op/s 2024-02-10T06:44:44.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:43 smithi029 ceph-mon[27077]: pgmap v245: 97 pgs: 97 active+clean; 42 MiB data, 96 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 3.5 MiB/s wr, 192 op/s 2024-02-10T06:44:44.602 INFO:teuthology.orchestra.run.smithi008.stdout: 5 3647 38.64 MB/sec warmup 5 sec latency 51.235 ms 2024-02-10T06:44:44.918 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:44 smithi008 ceph-mon[24646]: pgmap v246: 97 pgs: 97 active+clean; 109 MiB data, 137 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 9.5 MiB/s wr, 442 op/s 2024-02-10T06:44:45.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:44 smithi029 ceph-mon[27077]: pgmap v246: 97 pgs: 97 active+clean; 109 MiB data, 137 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 9.5 MiB/s wr, 442 op/s 2024-02-10T06:44:45.602 INFO:teuthology.orchestra.run.smithi008.stdout: 5 4333 33.73 MB/sec warmup 6 sec latency 46.872 ms 2024-02-10T06:44:46.603 INFO:teuthology.orchestra.run.smithi008.stdout: 5 5434 32.50 MB/sec warmup 7 sec latency 13.606 ms 2024-02-10T06:44:47.177 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:46 smithi008 ceph-mon[24646]: pgmap v247: 97 pgs: 97 active+clean; 109 MiB data, 137 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 9.5 MiB/s wr, 439 op/s 2024-02-10T06:44:47.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:46 smithi029 ceph-mon[27077]: pgmap v247: 97 pgs: 97 active+clean; 109 MiB data, 137 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 9.5 MiB/s wr, 439 op/s 2024-02-10T06:44:47.603 INFO:teuthology.orchestra.run.smithi008.stdout: 5 6238 30.87 MB/sec warmup 8 sec latency 18.859 ms 2024-02-10T06:44:48.603 INFO:teuthology.orchestra.run.smithi008.stdout: 5 7213 32.20 MB/sec warmup 9 sec latency 25.109 ms 2024-02-10T06:44:49.087 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-10T06:44:49.192 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:48 smithi008 ceph-mon[24646]: pgmap v248: 97 pgs: 97 active+clean; 129 MiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 13 MiB/s wr, 743 op/s 2024-02-10T06:44:49.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:48 smithi029 ceph-mon[27077]: pgmap v248: 97 pgs: 97 active+clean; 129 MiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 13 MiB/s wr, 743 op/s 2024-02-10T06:44:49.603 INFO:teuthology.orchestra.run.smithi008.stdout: 5 7906 29.84 MB/sec warmup 10 sec latency 26.032 ms 2024-02-10T06:44:50.603 INFO:teuthology.orchestra.run.smithi008.stdout: 5 8714 29.27 MB/sec warmup 11 sec latency 109.330 ms 2024-02-10T06:44:51.011 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:50 smithi008 ceph-mon[24646]: pgmap v249: 97 pgs: 97 active+clean; 142 MiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 16 MiB/s wr, 978 op/s 2024-02-10T06:44:51.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:50 smithi029 ceph-mon[27077]: pgmap v249: 97 pgs: 97 active+clean; 142 MiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 16 MiB/s wr, 978 op/s 2024-02-10T06:44:51.603 INFO:teuthology.orchestra.run.smithi008.stdout: 5 9719 28.40 MB/sec warmup 12 sec latency 30.680 ms 2024-02-10T06:44:52.603 INFO:teuthology.orchestra.run.smithi008.stdout: 5 10542 28.81 MB/sec warmup 13 sec latency 260.248 ms 2024-02-10T06:44:53.036 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:52 smithi008 ceph-mon[24646]: pgmap v250: 97 pgs: 97 active+clean; 155 MiB data, 343 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 18 MiB/s wr, 1.23k op/s 2024-02-10T06:44:53.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:52 smithi029 ceph-mon[27077]: pgmap v250: 97 pgs: 97 active+clean; 155 MiB data, 343 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 18 MiB/s wr, 1.23k op/s 2024-02-10T06:44:53.603 INFO:teuthology.orchestra.run.smithi008.stdout: 5 11183 27.98 MB/sec warmup 14 sec latency 24.686 ms 2024-02-10T06:44:54.603 INFO:teuthology.orchestra.run.smithi008.stdout: 5 11958 27.20 MB/sec warmup 15 sec latency 12.241 ms 2024-02-10T06:44:55.045 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:54 smithi008 ceph-mon[24646]: pgmap v251: 97 pgs: 97 active+clean; 150 MiB data, 405 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 17 MiB/s wr, 1.16k op/s 2024-02-10T06:44:55.116 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph mgr module enable nfs --force' 2024-02-10T06:44:55.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:54 smithi029 ceph-mon[27077]: pgmap v251: 97 pgs: 97 active+clean; 150 MiB data, 405 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 17 MiB/s wr, 1.16k op/s 2024-02-10T06:44:55.603 INFO:teuthology.orchestra.run.smithi008.stdout: 5 12994 26.54 MB/sec warmup 16 sec latency 17.939 ms 2024-02-10T06:44:56.604 INFO:teuthology.orchestra.run.smithi008.stdout: 5 14006 27.37 MB/sec warmup 17 sec latency 17.403 ms 2024-02-10T06:44:57.053 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:56 smithi008 ceph-mon[24646]: pgmap v252: 97 pgs: 97 active+clean; 150 MiB data, 405 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 11 MiB/s wr, 904 op/s 2024-02-10T06:44:57.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:56 smithi029 ceph-mon[27077]: pgmap v252: 97 pgs: 97 active+clean; 150 MiB data, 405 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 11 MiB/s wr, 904 op/s 2024-02-10T06:44:57.603 INFO:teuthology.orchestra.run.smithi008.stdout: 5 14672 27.07 MB/sec warmup 18 sec latency 26.929 ms 2024-02-10T06:44:58.603 INFO:teuthology.orchestra.run.smithi008.stdout: 5 15321 26.08 MB/sec warmup 19 sec latency 18.846 ms 2024-02-10T06:44:59.062 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:58 smithi008 ceph-mon[24646]: pgmap v253: 97 pgs: 97 active+clean; 156 MiB data, 484 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 14 MiB/s wr, 1.22k op/s 2024-02-10T06:44:59.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:58 smithi029 ceph-mon[27077]: pgmap v253: 97 pgs: 97 active+clean; 156 MiB data, 484 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 14 MiB/s wr, 1.22k op/s 2024-02-10T06:44:59.603 INFO:teuthology.orchestra.run.smithi008.stdout: 5 16372 25.89 MB/sec warmup 20 sec latency 17.005 ms 2024-02-10T06:45:00.008 INFO:teuthology.orchestra.run.smithi008.stderr:module 'nfs' is already enabled 2024-02-10T06:45:00.066 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:44:59 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/396464419' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-02-10T06:45:00.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:44:59 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/396464419' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-02-10T06:45:00.603 INFO:teuthology.orchestra.run.smithi008.stdout: 5 17434 26.60 MB/sec warmup 21 sec latency 19.028 ms 2024-02-10T06:45:01.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:00 smithi008 ceph-mon[24646]: pgmap v254: 97 pgs: 97 active+clean; 162 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 1.08k op/s 2024-02-10T06:45:01.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:00 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/396464419' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-02-10T06:45:01.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:00 smithi008 ceph-mon[24646]: mgrmap e21: smithi008.pxifbt(active, since 6m), standbys: smithi029.fuvrtf 2024-02-10T06:45:01.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:00 smithi029 ceph-mon[27077]: pgmap v254: 97 pgs: 97 active+clean; 162 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 1.08k op/s 2024-02-10T06:45:01.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:00 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/396464419' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-02-10T06:45:01.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:00 smithi029 ceph-mon[27077]: mgrmap e21: smithi008.pxifbt(active, since 6m), standbys: smithi029.fuvrtf 2024-02-10T06:45:01.605 INFO:teuthology.orchestra.run.smithi008.stdout: 5 18246 26.50 MB/sec warmup 22 sec latency 22.887 ms 2024-02-10T06:45:01.636 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-10T06:45:02.603 INFO:teuthology.orchestra.run.smithi008.stdout: 5 18916 25.85 MB/sec warmup 23 sec latency 14.613 ms 2024-02-10T06:45:03.080 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:02 smithi008 ceph-mon[24646]: pgmap v255: 97 pgs: 97 active+clean; 171 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 12 MiB/s wr, 1.13k op/s 2024-02-10T06:45:03.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:02 smithi029 ceph-mon[27077]: pgmap v255: 97 pgs: 97 active+clean; 171 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 12 MiB/s wr, 1.13k op/s 2024-02-10T06:45:03.604 INFO:teuthology.orchestra.run.smithi008.stdout: 5 19908 25.56 MB/sec warmup 24 sec latency 29.134 ms 2024-02-10T06:45:04.604 INFO:teuthology.orchestra.run.smithi008.stdout: 5 20870 25.90 MB/sec warmup 25 sec latency 26.552 ms 2024-02-10T06:45:04.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:04 smithi008 ceph-mon[24646]: pgmap v256: 97 pgs: 97 active+clean; 179 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 13 MiB/s wr, 1.08k op/s 2024-02-10T06:45:05.083 INFO:teuthology.orchestra.run.smithi008.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:45:05.136 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:04 smithi029 ceph-mon[27077]: pgmap v256: 97 pgs: 97 active+clean; 179 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 13 MiB/s wr, 1.08k op/s 2024-02-10T06:45:05.604 INFO:teuthology.orchestra.run.smithi008.stdout: 5 21662 26.06 MB/sec warmup 26 sec latency 26.603 ms 2024-02-10T06:45:06.152 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-10T06:45:06.153 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T06:45:06.153 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-02-10T06:45:06.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:06 smithi008 ceph-mon[24646]: from='client.24395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:06.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:06 smithi008 ceph-mon[24646]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:45:06.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:06 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:45:06.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:06 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:45:06.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:06 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:45:06.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:06 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:45:06.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:06 smithi029 ceph-mon[27077]: from='client.24395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:06.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:06 smithi029 ceph-mon[27077]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:45:06.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:06 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:45:06.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:06 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:45:06.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:06 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:45:06.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:06 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:45:06.604 INFO:teuthology.orchestra.run.smithi008.stdout: 5 22334 25.37 MB/sec warmup 27 sec latency 27.367 ms 2024-02-10T06:45:07.525 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:07 smithi008 ceph-mon[24646]: pgmap v257: 97 pgs: 97 active+clean; 179 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 970 op/s 2024-02-10T06:45:07.525 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:07 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:45:07.525 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:07 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:45:07.525 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:07 smithi008 ceph-mon[24646]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:45:07.605 INFO:teuthology.orchestra.run.smithi008.stdout: 5 23164 25.21 MB/sec warmup 28 sec latency 61.874 ms 2024-02-10T06:45:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:07 smithi029 ceph-mon[27077]: pgmap v257: 97 pgs: 97 active+clean; 179 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 970 op/s 2024-02-10T06:45:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:07 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:45:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:07 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:45:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:07 smithi029 ceph-mon[27077]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:45:08.605 INFO:teuthology.orchestra.run.smithi008.stdout: 5 24041 25.06 MB/sec warmup 29 sec latency 19.779 ms 2024-02-10T06:45:09.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:09 smithi008 ceph-mon[24646]: pgmap v258: 97 pgs: 97 active+clean; 191 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 14 MiB/s wr, 1.29k op/s 2024-02-10T06:45:09.605 INFO:teuthology.orchestra.run.smithi008.stdout: 5 25053 25.68 MB/sec warmup 30 sec latency 12.764 ms 2024-02-10T06:45:09.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:09 smithi029 ceph-mon[27077]: pgmap v258: 97 pgs: 97 active+clean; 191 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 14 MiB/s wr, 1.29k op/s 2024-02-10T06:45:10.605 INFO:teuthology.orchestra.run.smithi008.stdout: 5 25569 25.03 MB/sec warmup 31 sec latency 18.835 ms 2024-02-10T06:45:11.063 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:45:11.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:11 smithi008 ceph-mon[24646]: pgmap v259: 97 pgs: 97 active+clean; 190 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-02-10T06:45:11.605 INFO:teuthology.orchestra.run.smithi008.stdout: 5 26173 24.70 MB/sec warmup 32 sec latency 23.672 ms 2024-02-10T06:45:11.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:11 smithi029 ceph-mon[27077]: pgmap v259: 97 pgs: 97 active+clean; 190 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-02-10T06:45:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:45:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (5m) 34s ago 7m 24.7M - ba2b418f427c d582fd72ee55 2024-02-10T06:45:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (7m) 34s ago 7m 6405k - 17.2.0 e1d6a67b021e 5fa2e5bdd7cc 2024-02-10T06:45:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (6m) 36s ago 6m 6405k - 17.2.0 e1d6a67b021e 7527ea77b5cc 2024-02-10T06:45:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (5m) 34s ago 6m 54.5M - 8.3.5 dad864ee21e9 c9887c4d30ec 2024-02-10T06:45:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (65s) 34s ago 64s 15.0M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:45:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (62s) 36s ago 62s 12.0M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:45:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:9283 running (8m) 34s ago 8m 460M - 17.2.0 e1d6a67b021e b0d0b31e96f4 2024-02-10T06:45:12.094 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283 running (6m) 36s ago 6m 414M - 17.2.0 e1d6a67b021e 4847b9108037 2024-02-10T06:45:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (8m) 34s ago 8m 45.6M 2048M 17.2.0 e1d6a67b021e 061312907e4b 2024-02-10T06:45:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (6m) 36s ago 6m 39.9M 2048M 17.2.0 e1d6a67b021e a790a61584bf 2024-02-10T06:45:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (45s) 34s ago 44s 51.0M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:45:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (39s) 36s ago 39s 46.7M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:45:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (6m) 34s ago 6m 17.3M - 1dbe0e931976 3c06441739f8 2024-02-10T06:45:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (6m) 36s ago 6m 17.0M - 1dbe0e931976 6e5039760c8a 2024-02-10T06:45:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (5m) 34s ago 5m 45.5M 3217M 17.2.0 e1d6a67b021e 139b7a92bfc4 2024-02-10T06:45:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (5m) 34s ago 5m 43.5M 3217M 17.2.0 e1d6a67b021e 0f1119e17a52 2024-02-10T06:45:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (4m) 34s ago 4m 43.9M 3217M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:45:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (4m) 34s ago 4m 41.5M 3217M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:45:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (3m) 36s ago 3m 41.9M 3985M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:45:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (3m) 36s ago 3m 41.2M 3985M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:45:12.095 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (3m) 36s ago 3m 38.6M 3985M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:45:12.096 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (2m) 36s ago 2m 39.6M 3985M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:45:12.096 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (6m) 34s ago 6m 63.2M - 514e6a882f6e eedfd13fcab7 2024-02-10T06:45:12.546 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:12 smithi008 ceph-mon[24646]: from='client.24399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:12.546 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:12 smithi008 ceph-mon[24646]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:12.605 INFO:teuthology.orchestra.run.smithi008.stdout: 5 26874 24.34 MB/sec warmup 33 sec latency 47.285 ms 2024-02-10T06:45:12.608 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:12 smithi029 ceph-mon[27077]: from='client.24399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:12.608 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:12 smithi029 ceph-mon[27077]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:12.750 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:45:12.751 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:45:12.751 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:45:12.751 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:45:12.751 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:45:12.751 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:45:12.751 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:45:12.751 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:45:12.751 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:45:12.751 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:45:12.751 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:45:12.751 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:45:12.751 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:45:12.751 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:45:12.752 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-02-10T06:45:12.752 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:45:12.752 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:45:13.283 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:45:13.283 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T06:45:13.283 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:45:13.283 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-02-10T06:45:13.283 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "", 2024-02-10T06:45:13.283 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 image" 2024-02-10T06:45:13.283 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:45:13.554 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:13 smithi008 ceph-mon[24646]: pgmap v260: 97 pgs: 97 active+clean; 202 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 12 MiB/s wr, 1.20k op/s 2024-02-10T06:45:13.554 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:13 smithi008 ceph-mon[24646]: from='client.24405 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:13.554 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:13 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/928371157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:45:13.605 INFO:teuthology.orchestra.run.smithi008.stdout: 5 27541 24.16 MB/sec warmup 34 sec latency 23.176 ms 2024-02-10T06:45:13.624 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:13 smithi029 ceph-mon[27077]: pgmap v260: 97 pgs: 97 active+clean; 202 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 12 MiB/s wr, 1.20k op/s 2024-02-10T06:45:13.624 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:13 smithi029 ceph-mon[27077]: from='client.24405 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:13.624 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:13 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/928371157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:45:13.951 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:45:14.560 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:14 smithi008 ceph-mon[24646]: from='client.24413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:14.560 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:14 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/2580118045' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:45:14.605 INFO:teuthology.orchestra.run.smithi008.stdout: 5 28341 24.42 MB/sec warmup 35 sec latency 22.427 ms 2024-02-10T06:45:14.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:14 smithi029 ceph-mon[27077]: from='client.24413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:14.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:14 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/2580118045' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:45:15.564 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:15 smithi008 ceph-mon[24646]: pgmap v261: 97 pgs: 97 active+clean; 200 MiB data, 796 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-02-10T06:45:15.605 INFO:teuthology.orchestra.run.smithi008.stdout: 5 28921 24.16 MB/sec warmup 36 sec latency 16.402 ms 2024-02-10T06:45:15.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:15 smithi029 ceph-mon[27077]: pgmap v261: 97 pgs: 97 active+clean; 200 MiB data, 796 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-02-10T06:45:16.605 INFO:teuthology.orchestra.run.smithi008.stdout: 5 29688 23.90 MB/sec warmup 37 sec latency 17.369 ms 2024-02-10T06:45:17.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:17 smithi008 ceph-mon[24646]: pgmap v262: 97 pgs: 97 active+clean; 200 MiB data, 796 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 835 op/s 2024-02-10T06:45:17.605 INFO:teuthology.orchestra.run.smithi008.stdout: 5 30415 23.66 MB/sec warmup 38 sec latency 73.633 ms 2024-02-10T06:45:17.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:17 smithi029 ceph-mon[27077]: pgmap v262: 97 pgs: 97 active+clean; 200 MiB data, 796 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 835 op/s 2024-02-10T06:45:18.605 INFO:teuthology.orchestra.run.smithi008.stdout: 5 31100 23.52 MB/sec warmup 39 sec latency 26.467 ms 2024-02-10T06:45:19.595 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:19 smithi008 ceph-mon[24646]: pgmap v263: 97 pgs: 97 active+clean; 213 MiB data, 879 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 1.14k op/s 2024-02-10T06:45:19.605 INFO:teuthology.orchestra.run.smithi008.stdout: 5 31954 23.81 MB/sec warmup 40 sec latency 23.605 ms 2024-02-10T06:45:19.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:19 smithi029 ceph-mon[27077]: pgmap v263: 97 pgs: 97 active+clean; 213 MiB data, 879 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 1.14k op/s 2024-02-10T06:45:20.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 32530 23.56 MB/sec warmup 41 sec latency 26.635 ms 2024-02-10T06:45:21.453 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:21 smithi008 ceph-mon[24646]: pgmap v264: 97 pgs: 97 active+clean; 212 MiB data, 890 MiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 9.8 MiB/s wr, 876 op/s 2024-02-10T06:45:21.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 33115 23.30 MB/sec warmup 42 sec latency 34.535 ms 2024-02-10T06:45:21.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:21 smithi029 ceph-mon[27077]: pgmap v264: 97 pgs: 97 active+clean; 212 MiB data, 890 MiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 9.8 MiB/s wr, 876 op/s 2024-02-10T06:45:22.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 33872 23.13 MB/sec warmup 43 sec latency 29.682 ms 2024-02-10T06:45:23.473 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:23 smithi008 ceph-mon[24646]: pgmap v265: 97 pgs: 97 active+clean; 224 MiB data, 927 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-02-10T06:45:23.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 34783 23.14 MB/sec warmup 44 sec latency 15.851 ms 2024-02-10T06:45:23.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:23 smithi029 ceph-mon[27077]: pgmap v265: 97 pgs: 97 active+clean; 224 MiB data, 927 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-02-10T06:45:24.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 35602 23.34 MB/sec warmup 45 sec latency 17.094 ms 2024-02-10T06:45:24.987 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-02-10T06:45:24.987 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T06:45:24.987 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T06:45:24.987 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]: dispatch 2024-02-10T06:45:24.987 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:24 smithi008 ceph-mon[24646]: pgmap v266: 97 pgs: 97 active+clean; 224 MiB data, 965 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 898 op/s 2024-02-10T06:45:24.987 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:45:24.987 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:45:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-02-10T06:45:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-10T06:45:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T06:45:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]: dispatch 2024-02-10T06:45:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:24 smithi029 ceph-mon[27077]: pgmap v266: 97 pgs: 97 active+clean; 224 MiB data, 965 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 898 op/s 2024-02-10T06:45:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:45:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:45:25.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 36264 23.15 MB/sec warmup 46 sec latency 21.811 ms 2024-02-10T06:45:25.991 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:25 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-02-10T06:45:25.991 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:25 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-10T06:45:25.991 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:25 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-02-10T06:45:25.991 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:25 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-02-10T06:45:25.991 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:25 smithi008 ceph-mon[24646]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T06:45:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:25 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-02-10T06:45:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:25 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-10T06:45:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:25 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-02-10T06:45:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:25 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-02-10T06:45:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:25 smithi029 ceph-mon[27077]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T06:45:26.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 37059 23.01 MB/sec warmup 47 sec latency 18.316 ms 2024-02-10T06:45:26.996 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:26 smithi008 ceph-mon[24646]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T06:45:26.996 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:26 smithi008 ceph-mon[24646]: pgmap v269: 97 pgs: 97 active+clean; 224 MiB data, 965 MiB used, 714 GiB / 715 GiB avail; 7.6 KiB/s rd, 8.0 MiB/s wr, 696 op/s 2024-02-10T06:45:27.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:26 smithi029 ceph-mon[27077]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T06:45:27.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:26 smithi029 ceph-mon[27077]: pgmap v269: 97 pgs: 97 active+clean; 224 MiB data, 965 MiB used, 714 GiB / 715 GiB avail; 7.6 KiB/s rd, 8.0 MiB/s wr, 696 op/s 2024-02-10T06:45:27.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 37867 22.89 MB/sec warmup 48 sec latency 38.257 ms 2024-02-10T06:45:28.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 38620 23.02 MB/sec warmup 49 sec latency 29.891 ms 2024-02-10T06:45:29.010 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:28 smithi008 ceph-mon[24646]: pgmap v270: 97 pgs: 97 active+clean; 231 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 1.09k op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T06:45:29.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:28 smithi029 ceph-mon[27077]: pgmap v270: 97 pgs: 97 active+clean; 231 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 1.09k op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T06:45:29.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 39276 23.00 MB/sec warmup 50 sec latency 20.021 ms 2024-02-10T06:45:30.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 39792 22.74 MB/sec warmup 51 sec latency 31.758 ms 2024-02-10T06:45:31.023 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:31 smithi008 ceph-mon[24646]: pgmap v271: 97 pgs: 97 active+clean; 236 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.9 KiB/s rd, 10 MiB/s wr, 842 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T06:45:31.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:31 smithi029 ceph-mon[27077]: pgmap v271: 97 pgs: 97 active+clean; 236 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.9 KiB/s rd, 10 MiB/s wr, 842 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T06:45:31.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 40383 22.54 MB/sec warmup 52 sec latency 21.671 ms 2024-02-10T06:45:32.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 41049 22.39 MB/sec warmup 53 sec latency 26.863 ms 2024-02-10T06:45:33.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:33 smithi008 ceph-mon[24646]: pgmap v272: 97 pgs: 97 active+clean; 247 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 973 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T06:45:33.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:33 smithi029 ceph-mon[27077]: pgmap v272: 97 pgs: 97 active+clean; 247 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 10 MiB/s wr, 973 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T06:45:33.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 41677 22.25 MB/sec warmup 54 sec latency 23.058 ms 2024-02-10T06:45:34.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 42237 22.27 MB/sec warmup 55 sec latency 35.503 ms 2024-02-10T06:45:35.052 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:34 smithi008 ceph-mon[24646]: pgmap v273: 97 pgs: 97 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 12 MiB/s wr, 1.06k op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T06:45:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:34 smithi029 ceph-mon[27077]: pgmap v273: 97 pgs: 97 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 12 MiB/s wr, 1.06k op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T06:45:35.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 42751 22.24 MB/sec warmup 56 sec latency 38.214 ms 2024-02-10T06:45:36.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 43273 22.01 MB/sec warmup 57 sec latency 20.174 ms 2024-02-10T06:45:37.063 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:37 smithi008 ceph-mon[24646]: pgmap v274: 97 pgs: 97 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 10 MiB/s wr, 953 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T06:45:37.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:37 smithi029 ceph-mon[27077]: pgmap v274: 97 pgs: 97 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 10 MiB/s wr, 953 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T06:45:37.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 43824 21.83 MB/sec warmup 58 sec latency 40.928 ms 2024-02-10T06:45:38.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 44707 21.75 MB/sec warmup 59 sec latency 12.695 ms 2024-02-10T06:45:39.185 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:39 smithi008 ceph-mon[24646]: pgmap v275: 97 pgs: 97 active+clean; 253 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.06k op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T06:45:39.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:39 smithi029 ceph-mon[27077]: pgmap v275: 97 pgs: 97 active+clean; 253 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.06k op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T06:45:39.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 45450 21.76 MB/sec warmup 60 sec latency 34.214 ms 2024-02-10T06:45:40.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 45926 21.75 MB/sec warmup 61 sec latency 63.776 ms 2024-02-10T06:45:41.201 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:41 smithi008 ceph-mon[24646]: pgmap v276: 97 pgs: 97 active+clean; 254 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 788 op/s 2024-02-10T06:45:41.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:41 smithi029 ceph-mon[27077]: pgmap v276: 97 pgs: 97 active+clean; 254 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 788 op/s 2024-02-10T06:45:41.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 46314 21.63 MB/sec warmup 62 sec latency 37.903 ms 2024-02-10T06:45:42.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 46747 21.41 MB/sec warmup 63 sec latency 24.701 ms 2024-02-10T06:45:43.216 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:43 smithi008 ceph-mon[24646]: pgmap v277: 97 pgs: 97 active+clean; 263 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.7 MiB/s wr, 952 op/s 2024-02-10T06:45:43.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:43 smithi029 ceph-mon[27077]: pgmap v277: 97 pgs: 97 active+clean; 263 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.7 MiB/s wr, 952 op/s 2024-02-10T06:45:43.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 47279 21.26 MB/sec warmup 64 sec latency 23.192 ms 2024-02-10T06:45:44.488 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:45:44.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 47899 21.12 MB/sec warmup 65 sec latency 26.721 ms 2024-02-10T06:45:45.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:45 smithi008 ceph-mon[24646]: pgmap v278: 97 pgs: 97 active+clean; 265 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 820 op/s 2024-02-10T06:45:45.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:45 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:45:45.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:45 smithi008 ceph-mon[24646]: Upgrade: Target is version 19.0.0-1361-g290212a1 (unknown) 2024-02-10T06:45:45.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:45 smithi008 ceph-mon[24646]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:45:45.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:45 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:45:45.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:45 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:45:45.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:45 smithi008 ceph-mon[24646]: Upgrade: Need to upgrade myself (mgr.smithi008.pxifbt) 2024-02-10T06:45:45.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:45 smithi008 ceph-mon[24646]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:45.234 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:45 smithi008 ceph-mon[24646]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 on smithi029 2024-02-10T06:45:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:45 smithi029 ceph-mon[27077]: pgmap v278: 97 pgs: 97 active+clean; 265 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 820 op/s 2024-02-10T06:45:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:45 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:45:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:45 smithi029 ceph-mon[27077]: Upgrade: Target is version 19.0.0-1361-g290212a1 (unknown) 2024-02-10T06:45:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:45 smithi029 ceph-mon[27077]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:45:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:45 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:45:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:45 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:45:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:45 smithi029 ceph-mon[27077]: Upgrade: Need to upgrade myself (mgr.smithi008.pxifbt) 2024-02-10T06:45:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:45 smithi029 ceph-mon[27077]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:45 smithi029 ceph-mon[27077]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 on smithi029 2024-02-10T06:45:45.505 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:45:45.505 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (6m) 67s ago 7m 24.7M - ba2b418f427c d582fd72ee55 2024-02-10T06:45:45.505 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (7m) 67s ago 7m 6405k - 17.2.0 e1d6a67b021e 5fa2e5bdd7cc 2024-02-10T06:45:45.506 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (7m) 70s ago 7m 6405k - 17.2.0 e1d6a67b021e 7527ea77b5cc 2024-02-10T06:45:45.506 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (6m) 67s ago 7m 54.5M - 8.3.5 dad864ee21e9 c9887c4d30ec 2024-02-10T06:45:45.506 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (99s) 67s ago 98s 15.0M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:45:45.506 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (96s) 70s ago 95s 12.0M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:45:45.506 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:9283 running (9m) 67s ago 9m 460M - 17.2.0 e1d6a67b021e b0d0b31e96f4 2024-02-10T06:45:45.506 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283 running (7m) 70s ago 7m 414M - 17.2.0 e1d6a67b021e 4847b9108037 2024-02-10T06:45:45.506 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (9m) 67s ago 9m 45.6M 2048M 17.2.0 e1d6a67b021e 061312907e4b 2024-02-10T06:45:45.506 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (6m) 70s ago 6m 39.9M 2048M 17.2.0 e1d6a67b021e a790a61584bf 2024-02-10T06:45:45.506 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (78s) 67s ago 78s 51.0M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:45:45.506 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (73s) 70s ago 72s 46.7M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:45:45.506 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (7m) 67s ago 7m 17.3M - 1dbe0e931976 3c06441739f8 2024-02-10T06:45:45.506 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (6m) 70s ago 6m 17.0M - 1dbe0e931976 6e5039760c8a 2024-02-10T06:45:45.506 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (6m) 67s ago 6m 45.5M 3217M 17.2.0 e1d6a67b021e 139b7a92bfc4 2024-02-10T06:45:45.506 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (5m) 67s ago 5m 43.5M 3217M 17.2.0 e1d6a67b021e 0f1119e17a52 2024-02-10T06:45:45.506 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (5m) 67s ago 5m 43.9M 3217M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:45:45.507 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (4m) 67s ago 4m 41.5M 3217M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:45:45.507 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (4m) 70s ago 4m 41.9M 3985M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:45:45.507 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (4m) 70s ago 4m 41.2M 3985M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:45:45.507 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (3m) 70s ago 3m 38.6M 3985M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:45:45.507 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (3m) 70s ago 3m 39.6M 3985M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:45:45.507 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (6m) 67s ago 6m 63.2M - 514e6a882f6e eedfd13fcab7 2024-02-10T06:45:45.606 INFO:teuthology.orchestra.run.smithi008.stdout: 5 48667 21.07 MB/sec warmup 66 sec latency 20.198 ms 2024-02-10T06:45:46.165 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:45:46.165 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:45:46.165 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:45:46.165 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:45:46.165 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:45:46.165 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:45:46.165 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:45:46.165 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:45:46.165 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:45:46.165 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:45:46.166 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:45:46.166 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:45:46.166 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:45:46.166 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:45:46.166 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-02-10T06:45:46.166 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:45:46.166 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:45:46.237 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:46 smithi008 ceph-mon[24646]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:46.237 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:46 smithi008 ceph-mon[24646]: from='client.24429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:46.237 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:46 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/2730247997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:45:46.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 49383 21.17 MB/sec warmup 67 sec latency 16.779 ms 2024-02-10T06:45:46.706 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:45:46.706 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:45:46.706 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:45:46.706 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-02-10T06:45:46.706 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "0/23 daemons upgraded", 2024-02-10T06:45:46.706 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 image on host smithi029" 2024-02-10T06:45:46.706 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:45:46.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:46 smithi029 ceph-mon[27077]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:46.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:46 smithi029 ceph-mon[27077]: from='client.24429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:46.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:46 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/2730247997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:45:47.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:47 smithi008 ceph-mon[24646]: pgmap v279: 97 pgs: 97 active+clean; 265 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 667 op/s 2024-02-10T06:45:47.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:47 smithi008 ceph-mon[24646]: from='client.24435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:47.339 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:45:47.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 49955 21.14 MB/sec warmup 68 sec latency 25.800 ms 2024-02-10T06:45:47.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:47 smithi029 ceph-mon[27077]: pgmap v279: 97 pgs: 97 active+clean; 265 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 667 op/s 2024-02-10T06:45:47.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:47 smithi029 ceph-mon[27077]: from='client.24435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:45:48.268 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:48 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/249289339' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:45:48.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 50361 20.96 MB/sec warmup 69 sec latency 43.094 ms 2024-02-10T06:45:48.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:48 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/249289339' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:45:49.279 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:49 smithi008 ceph-mon[24646]: pgmap v280: 97 pgs: 97 active+clean; 271 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.7 MiB/s wr, 941 op/s 2024-02-10T06:45:49.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 50962 20.82 MB/sec warmup 70 sec latency 21.454 ms 2024-02-10T06:45:49.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:49 smithi029 ceph-mon[27077]: pgmap v280: 97 pgs: 97 active+clean; 271 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.7 MiB/s wr, 941 op/s 2024-02-10T06:45:50.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 51564 20.71 MB/sec warmup 71 sec latency 100.206 ms 2024-02-10T06:45:51.296 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:51 smithi008 ceph-mon[24646]: pgmap v281: 97 pgs: 97 active+clean; 274 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.4 MiB/s wr, 746 op/s 2024-02-10T06:45:51.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 52389 20.72 MB/sec warmup 72 sec latency 18.826 ms 2024-02-10T06:45:51.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:51 smithi029 ceph-mon[27077]: pgmap v281: 97 pgs: 97 active+clean; 274 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.4 MiB/s wr, 746 op/s 2024-02-10T06:45:52.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 53238 20.90 MB/sec warmup 73 sec latency 20.369 ms 2024-02-10T06:45:53.311 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:53 smithi008 ceph-mon[24646]: pgmap v282: 97 pgs: 97 active+clean; 281 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.0 MiB/s wr, 911 op/s 2024-02-10T06:45:53.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 53892 20.83 MB/sec warmup 74 sec latency 24.348 ms 2024-02-10T06:45:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:53 smithi029 ceph-mon[27077]: pgmap v282: 97 pgs: 97 active+clean; 281 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.0 MiB/s wr, 911 op/s 2024-02-10T06:45:54.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 54486 20.70 MB/sec warmup 75 sec latency 29.891 ms 2024-02-10T06:45:55.074 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:54 smithi008 ceph-mon[24646]: pgmap v283: 97 pgs: 97 active+clean; 280 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.7 MiB/s wr, 765 op/s 2024-02-10T06:45:55.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:54 smithi029 ceph-mon[27077]: pgmap v283: 97 pgs: 97 active+clean; 280 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.7 MiB/s wr, 765 op/s 2024-02-10T06:45:55.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 55239 20.62 MB/sec warmup 76 sec latency 28.105 ms 2024-02-10T06:45:56.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 56156 20.74 MB/sec warmup 77 sec latency 13.272 ms 2024-02-10T06:45:57.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:57 smithi008 ceph-mon[24646]: pgmap v284: 97 pgs: 97 active+clean; 280 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 636 op/s 2024-02-10T06:45:57.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:57 smithi029 ceph-mon[27077]: pgmap v284: 97 pgs: 97 active+clean; 280 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 636 op/s 2024-02-10T06:45:57.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 56885 20.85 MB/sec warmup 78 sec latency 24.106 ms 2024-02-10T06:45:58.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 57468 20.73 MB/sec warmup 79 sec latency 29.159 ms 2024-02-10T06:45:59.184 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:45:59 smithi008 ceph-mon[24646]: pgmap v285: 97 pgs: 97 active+clean; 295 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 1k op/s 2024-02-10T06:45:59.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:45:59 smithi029 ceph-mon[27077]: pgmap v285: 97 pgs: 97 active+clean; 295 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 1k op/s 2024-02-10T06:45:59.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 58008 20.60 MB/sec warmup 80 sec latency 119.460 ms 2024-02-10T06:46:00.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 58438 20.44 MB/sec warmup 81 sec latency 196.500 ms 2024-02-10T06:46:01.225 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:01 smithi008 ceph-mon[24646]: pgmap v286: 97 pgs: 97 active+clean; 296 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 861 op/s 2024-02-10T06:46:01.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:01 smithi029 ceph-mon[27077]: pgmap v286: 97 pgs: 97 active+clean; 296 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 861 op/s 2024-02-10T06:46:01.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 59094 20.40 MB/sec warmup 82 sec latency 35.403 ms 2024-02-10T06:46:02.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 59957 20.53 MB/sec warmup 83 sec latency 19.156 ms 2024-02-10T06:46:03.246 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:03 smithi008 ceph-mon[24646]: pgmap v287: 97 pgs: 97 active+clean; 298 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.9 MiB/s wr, 979 op/s 2024-02-10T06:46:03.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:03 smithi029 ceph-mon[27077]: pgmap v287: 97 pgs: 97 active+clean; 298 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.9 MiB/s wr, 979 op/s 2024-02-10T06:46:03.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 60617 20.54 MB/sec warmup 84 sec latency 22.738 ms 2024-02-10T06:46:04.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 61186 20.44 MB/sec warmup 85 sec latency 115.854 ms 2024-02-10T06:46:05.056 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:04 smithi008 ceph-mon[24646]: pgmap v288: 97 pgs: 97 active+clean; 300 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.8 MiB/s wr, 840 op/s 2024-02-10T06:46:05.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:04 smithi029 ceph-mon[27077]: pgmap v288: 97 pgs: 97 active+clean; 300 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.8 MiB/s wr, 840 op/s 2024-02-10T06:46:05.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 61831 20.34 MB/sec warmup 86 sec latency 28.454 ms 2024-02-10T06:46:06.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 62598 20.32 MB/sec warmup 87 sec latency 26.376 ms 2024-02-10T06:46:07.067 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:07 smithi008 ceph-mon[24646]: pgmap v289: 97 pgs: 97 active+clean; 300 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 753 op/s 2024-02-10T06:46:07.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:07 smithi029 ceph-mon[27077]: pgmap v289: 97 pgs: 97 active+clean; 300 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 753 op/s 2024-02-10T06:46:07.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 63292 20.33 MB/sec warmup 88 sec latency 88.109 ms 2024-02-10T06:46:08.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 63947 20.41 MB/sec warmup 89 sec latency 22.791 ms 2024-02-10T06:46:09.107 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:09 smithi008 ceph-mon[24646]: pgmap v290: 97 pgs: 97 active+clean; 311 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 1.10k op/s 2024-02-10T06:46:09.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:09 smithi029 ceph-mon[27077]: pgmap v290: 97 pgs: 97 active+clean; 311 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 1.10k op/s 2024-02-10T06:46:09.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 64465 20.30 MB/sec warmup 90 sec latency 37.531 ms 2024-02-10T06:46:10.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 64957 20.19 MB/sec warmup 91 sec latency 28.970 ms 2024-02-10T06:46:11.124 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:11 smithi008 ceph-mon[24646]: pgmap v291: 97 pgs: 97 active+clean; 313 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 814 op/s 2024-02-10T06:46:11.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:11 smithi029 ceph-mon[27077]: pgmap v291: 97 pgs: 97 active+clean; 313 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 814 op/s 2024-02-10T06:46:11.607 INFO:teuthology.orchestra.run.smithi008.stdout: 5 65488 20.07 MB/sec warmup 92 sec latency 33.507 ms 2024-02-10T06:46:12.614 INFO:teuthology.orchestra.run.smithi008.stdout: 5 66136 20.04 MB/sec warmup 93 sec latency 20.368 ms 2024-02-10T06:46:13.024 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:13 smithi029 ceph-mon[27077]: pgmap v292: 97 pgs: 97 active+clean; 319 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 9.2 MiB/s wr, 913 op/s 2024-02-10T06:46:13.109 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:13 smithi008 ceph-mon[24646]: pgmap v292: 97 pgs: 97 active+clean; 319 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 9.2 MiB/s wr, 913 op/s 2024-02-10T06:46:13.614 INFO:teuthology.orchestra.run.smithi008.stdout: 5 66707 20.01 MB/sec warmup 94 sec latency 29.006 ms 2024-02-10T06:46:14.614 INFO:teuthology.orchestra.run.smithi008.stdout: 5 67153 19.97 MB/sec warmup 95 sec latency 64.583 ms 2024-02-10T06:46:14.918 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:14 smithi008 ceph-mon[24646]: pgmap v293: 97 pgs: 97 active+clean; 318 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 9.3 MiB/s wr, 894 op/s 2024-02-10T06:46:15.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:14 smithi029 ceph-mon[27077]: pgmap v293: 97 pgs: 97 active+clean; 318 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 9.3 MiB/s wr, 894 op/s 2024-02-10T06:46:15.614 INFO:teuthology.orchestra.run.smithi008.stdout: 5 67692 19.95 MB/sec warmup 96 sec latency 27.492 ms 2024-02-10T06:46:16.614 INFO:teuthology.orchestra.run.smithi008.stdout: 5 68134 19.84 MB/sec warmup 97 sec latency 28.485 ms 2024-02-10T06:46:17.277 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:17 smithi008 ceph-mon[24646]: pgmap v294: 97 pgs: 97 active+clean; 318 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 821 op/s 2024-02-10T06:46:17.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:17 smithi029 ceph-mon[27077]: pgmap v294: 97 pgs: 97 active+clean; 318 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 821 op/s 2024-02-10T06:46:17.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 68594 19.73 MB/sec warmup 98 sec latency 80.795 ms 2024-02-10T06:46:17.873 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:46:18.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 69141 19.65 MB/sec warmup 99 sec latency 35.005 ms 2024-02-10T06:46:18.820 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:46:18.820 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (6m) 101s ago 8m 24.7M - ba2b418f427c d582fd72ee55 2024-02-10T06:46:18.820 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (8m) 101s ago 8m 6405k - 17.2.0 e1d6a67b021e 5fa2e5bdd7cc 2024-02-10T06:46:18.820 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (7m) 103s ago 7m 6405k - 17.2.0 e1d6a67b021e 7527ea77b5cc 2024-02-10T06:46:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (6m) 101s ago 8m 54.5M - 8.3.5 dad864ee21e9 c9887c4d30ec 2024-02-10T06:46:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (2m) 101s ago 2m 15.0M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:46:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (2m) 103s ago 2m 12.0M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:46:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:9283 running (9m) 101s ago 9m 460M - 17.2.0 e1d6a67b021e b0d0b31e96f4 2024-02-10T06:46:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283 running (7m) 103s ago 7m 414M - 17.2.0 e1d6a67b021e 4847b9108037 2024-02-10T06:46:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (9m) 101s ago 9m 45.6M 2048M 17.2.0 e1d6a67b021e 061312907e4b 2024-02-10T06:46:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (7m) 103s ago 7m 39.9M 2048M 17.2.0 e1d6a67b021e a790a61584bf 2024-02-10T06:46:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (112s) 101s ago 111s 51.0M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:46:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (106s) 103s ago 106s 46.7M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:46:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (8m) 101s ago 8m 17.3M - 1dbe0e931976 3c06441739f8 2024-02-10T06:46:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (7m) 103s ago 7m 17.0M - 1dbe0e931976 6e5039760c8a 2024-02-10T06:46:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (6m) 101s ago 6m 45.5M 3217M 17.2.0 e1d6a67b021e 139b7a92bfc4 2024-02-10T06:46:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (6m) 101s ago 6m 43.5M 3217M 17.2.0 e1d6a67b021e 0f1119e17a52 2024-02-10T06:46:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (5m) 101s ago 5m 43.9M 3217M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:46:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (5m) 101s ago 5m 41.5M 3217M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:46:18.821 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (5m) 103s ago 5m 41.9M 3985M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:46:18.822 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (4m) 103s ago 4m 41.2M 3985M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:46:18.822 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (4m) 103s ago 4m 38.6M 3985M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:46:18.822 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (3m) 103s ago 3m 39.6M 3985M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:46:18.822 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (7m) 101s ago 7m 63.2M - 514e6a882f6e eedfd13fcab7 2024-02-10T06:46:19.066 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:19 smithi008 ceph-mon[24646]: from='client.24443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:46:19.066 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:19 smithi008 ceph-mon[24646]: pgmap v295: 97 pgs: 97 active+clean; 331 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 1.12k op/s 2024-02-10T06:46:19.066 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:19 smithi008 ceph-mon[24646]: from='client.24447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:46:19.066 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:19 smithi008 ceph-mon[24646]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:46:19.311 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:46:19.311 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:46:19.311 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:46:19.312 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:46:19.312 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:46:19.312 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:46:19.312 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:46:19.312 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:46:19.312 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:46:19.312 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:46:19.312 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:46:19.312 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:46:19.312 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:46:19.312 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:46:19.312 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-02-10T06:46:19.312 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:46:19.313 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:46:19.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:19 smithi029 ceph-mon[27077]: from='client.24443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:46:19.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:19 smithi029 ceph-mon[27077]: pgmap v295: 97 pgs: 97 active+clean; 331 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 1.12k op/s 2024-02-10T06:46:19.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:19 smithi029 ceph-mon[27077]: from='client.24447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:46:19.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:19 smithi029 ceph-mon[27077]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:46:19.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 69636 19.57 MB/sec warmup 100 sec latency 75.644 ms 2024-02-10T06:46:19.716 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:46:19.717 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:46:19.717 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:46:19.717 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-02-10T06:46:19.717 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "0/23 daemons upgraded", 2024-02-10T06:46:19.717 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 image on host smithi029" 2024-02-10T06:46:19.717 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:46:20.133 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:46:20.133 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:20 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1549005840' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:46:20.133 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:20 smithi008 ceph-mon[24646]: from='client.14684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:46:20.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:20 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1549005840' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:46:20.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:20 smithi029 ceph-mon[27077]: from='client.14684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:46:20.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 69959 19.46 MB/sec warmup 101 sec latency 223.262 ms 2024-02-10T06:46:21.282 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:21 smithi008 ceph-mon[24646]: pgmap v296: 97 pgs: 97 active+clean; 323 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 8.3 MiB/s wr, 813 op/s 2024-02-10T06:46:21.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:21 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/3090641363' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:46:21.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:21 smithi029 ceph-mon[27077]: pgmap v296: 97 pgs: 97 active+clean; 323 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 8.3 MiB/s wr, 813 op/s 2024-02-10T06:46:21.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:21 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/3090641363' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:46:21.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 70399 19.42 MB/sec warmup 102 sec latency 103.421 ms 2024-02-10T06:46:22.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 70694 19.35 MB/sec warmup 103 sec latency 179.385 ms 2024-02-10T06:46:23.185 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:23 smithi008 ceph-mon[24646]: pgmap v297: 97 pgs: 97 active+clean; 319 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.1 MiB/s wr, 871 op/s 2024-02-10T06:46:23.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:23 smithi029 ceph-mon[27077]: pgmap v297: 97 pgs: 97 active+clean; 319 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.1 MiB/s wr, 871 op/s 2024-02-10T06:46:23.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 71194 19.33 MB/sec warmup 104 sec latency 32.996 ms 2024-02-10T06:46:24.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 71665 19.24 MB/sec warmup 105 sec latency 99.124 ms 2024-02-10T06:46:24.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:46:24.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:24 smithi029 ceph-mon[27077]: pgmap v298: 97 pgs: 97 active+clean; 316 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 7.6 MiB/s wr, 719 op/s 2024-02-10T06:46:24.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:24 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:46:25.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:46:25.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:24 smithi008 ceph-mon[24646]: pgmap v298: 97 pgs: 97 active+clean; 316 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 7.6 MiB/s wr, 719 op/s 2024-02-10T06:46:25.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:24 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:46:25.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 72158 19.15 MB/sec warmup 106 sec latency 219.981 ms 2024-02-10T06:46:26.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 72874 19.11 MB/sec warmup 107 sec latency 21.551 ms 2024-02-10T06:46:27.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:27 smithi008 ceph-mon[24646]: pgmap v299: 97 pgs: 97 active+clean; 316 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.8 MiB/s wr, 556 op/s 2024-02-10T06:46:27.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:27 smithi029 ceph-mon[27077]: pgmap v299: 97 pgs: 97 active+clean; 316 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.8 MiB/s wr, 556 op/s 2024-02-10T06:46:27.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 73637 19.12 MB/sec warmup 108 sec latency 27.479 ms 2024-02-10T06:46:28.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 74270 19.17 MB/sec warmup 109 sec latency 100.824 ms 2024-02-10T06:46:29.436 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:29 smithi008 ceph-mon[24646]: pgmap v300: 97 pgs: 97 active+clean; 317 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 868 op/s 2024-02-10T06:46:29.436 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:29 smithi008 ceph-mon[24646]: Upgrade: Updating mgr.smithi029.fuvrtf 2024-02-10T06:46:29.437 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:29 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:46:29.437 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:29 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi029.fuvrtf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:46:29.437 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:29 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:46:29.437 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:29 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:46:29.437 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:29 smithi008 ceph-mon[24646]: Deploying daemon mgr.smithi029.fuvrtf on smithi029 2024-02-10T06:46:29.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 74911 19.17 MB/sec warmup 110 sec latency 25.337 ms 2024-02-10T06:46:29.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:29 smithi029 ceph-mon[27077]: pgmap v300: 97 pgs: 97 active+clean; 317 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 868 op/s 2024-02-10T06:46:29.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:29 smithi029 ceph-mon[27077]: Upgrade: Updating mgr.smithi029.fuvrtf 2024-02-10T06:46:29.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:29 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:46:29.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:29 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi029.fuvrtf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:46:29.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:29 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:46:29.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:29 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:46:29.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:29 smithi029 ceph-mon[27077]: Deploying daemon mgr.smithi029.fuvrtf on smithi029 2024-02-10T06:46:30.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 75301 19.07 MB/sec warmup 111 sec latency 119.321 ms 2024-02-10T06:46:31.498 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:31 smithi008 ceph-mon[24646]: pgmap v301: 97 pgs: 97 active+clean; 317 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.6 MiB/s wr, 622 op/s 2024-02-10T06:46:31.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 75703 18.98 MB/sec warmup 112 sec latency 203.657 ms 2024-02-10T06:46:31.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:31 smithi029 ceph-mon[27077]: pgmap v301: 97 pgs: 97 active+clean; 317 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.6 MiB/s wr, 622 op/s 2024-02-10T06:46:32.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 76411 18.94 MB/sec warmup 113 sec latency 25.104 ms 2024-02-10T06:46:33.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:33 smithi008 ceph-mon[24646]: pgmap v302: 97 pgs: 97 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 7.5 MiB/s wr, 772 op/s 2024-02-10T06:46:33.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 77185 18.97 MB/sec warmup 114 sec latency 87.120 ms 2024-02-10T06:46:33.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:33 smithi029 ceph-mon[27077]: pgmap v302: 97 pgs: 97 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 7.5 MiB/s wr, 772 op/s 2024-02-10T06:46:34.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 77654 18.96 MB/sec warmup 115 sec latency 135.363 ms 2024-02-10T06:46:35.068 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:34 smithi008 ceph-mon[24646]: pgmap v303: 97 pgs: 97 active+clean; 317 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 7.9 MiB/s wr, 709 op/s 2024-02-10T06:46:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:34 smithi029 ceph-mon[27077]: pgmap v303: 97 pgs: 97 active+clean; 317 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 7.9 MiB/s wr, 709 op/s 2024-02-10T06:46:35.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 78078 18.91 MB/sec warmup 116 sec latency 159.754 ms 2024-02-10T06:46:36.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 78535 18.85 MB/sec warmup 117 sec latency 33.414 ms 2024-02-10T06:46:37.281 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:37 smithi008 ceph-mon[24646]: pgmap v304: 97 pgs: 97 active+clean; 317 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.5 MiB/s wr, 628 op/s 2024-02-10T06:46:37.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:37 smithi029 ceph-mon[27077]: pgmap v304: 97 pgs: 97 active+clean; 317 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.5 MiB/s wr, 628 op/s 2024-02-10T06:46:37.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 79036 18.78 MB/sec warmup 118 sec latency 34.188 ms 2024-02-10T06:46:38.615 INFO:teuthology.orchestra.run.smithi008.stdout: 5 79866 18.77 MB/sec warmup 119 sec latency 21.734 ms 2024-02-10T06:46:39.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:39 smithi008 ceph-mon[24646]: pgmap v305: 97 pgs: 97 active+clean; 323 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 913 op/s 2024-02-10T06:46:39.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:39 smithi029 ceph-mon[27077]: pgmap v305: 97 pgs: 97 active+clean; 323 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 913 op/s 2024-02-10T06:46:40.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 81034 24.99 MB/sec execute 1 sec latency 486.306 ms 2024-02-10T06:46:41.166 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:41 smithi008 ceph-mon[24646]: pgmap v306: 97 pgs: 97 active+clean; 319 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 7.7 MiB/s wr, 638 op/s 2024-02-10T06:46:41.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:41 smithi029 ceph-mon[27077]: pgmap v306: 97 pgs: 97 active+clean; 319 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 7.7 MiB/s wr, 638 op/s 2024-02-10T06:46:41.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 81566 22.31 MB/sec execute 2 sec latency 122.225 ms 2024-02-10T06:46:42.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 82055 19.40 MB/sec execute 3 sec latency 126.262 ms 2024-02-10T06:46:43.176 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:43 smithi008 ceph-mon[24646]: pgmap v307: 97 pgs: 97 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.9 MiB/s wr, 830 op/s 2024-02-10T06:46:43.245 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:43 smithi029 ceph-mon[27077]: pgmap v307: 97 pgs: 97 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.9 MiB/s wr, 830 op/s 2024-02-10T06:46:43.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 82403 17.05 MB/sec execute 4 sec latency 40.877 ms 2024-02-10T06:46:44.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 82794 15.07 MB/sec execute 5 sec latency 43.553 ms 2024-02-10T06:46:44.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:44 smithi029 ceph-mon[27077]: pgmap v308: 97 pgs: 97 active+clean; 326 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.7 MiB/s wr, 779 op/s 2024-02-10T06:46:44.959 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:44 smithi008 ceph-mon[24646]: pgmap v308: 97 pgs: 97 active+clean; 326 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.7 MiB/s wr, 779 op/s 2024-02-10T06:46:45.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 83245 13.78 MB/sec execute 6 sec latency 42.890 ms 2024-02-10T06:46:46.479 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:46 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:46:46.479 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:46 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:46:46.479 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:46 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:46:46.479 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:46 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:46:46.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 83825 13.76 MB/sec execute 7 sec latency 82.190 ms 2024-02-10T06:46:46.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:46 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:46:46.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:46 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:46:46.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:46 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:46:46.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:46 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:46:47.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:47 smithi029 ceph-mon[27077]: pgmap v309: 97 pgs: 97 active+clean; 326 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.3 MiB/s wr, 703 op/s 2024-02-10T06:46:47.500 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:47 smithi008 ceph-mon[24646]: pgmap v309: 97 pgs: 97 active+clean; 326 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.3 MiB/s wr, 703 op/s 2024-02-10T06:46:47.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 84394 14.21 MB/sec execute 8 sec latency 30.397 ms 2024-02-10T06:46:48.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 84881 14.76 MB/sec execute 9 sec latency 139.074 ms 2024-02-10T06:46:49.543 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:49 smithi008 ceph-mon[24646]: pgmap v310: 97 pgs: 97 active+clean; 319 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.9 MiB/s wr, 910 op/s 2024-02-10T06:46:49.614 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:49 smithi029 ceph-mon[27077]: pgmap v310: 97 pgs: 97 active+clean; 319 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.9 MiB/s wr, 910 op/s 2024-02-10T06:46:49.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 85352 14.99 MB/sec execute 10 sec latency 29.923 ms 2024-02-10T06:46:50.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 85789 14.46 MB/sec execute 11 sec latency 56.206 ms 2024-02-10T06:46:50.662 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:46:51.402 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:51 smithi029 ceph-mon[27077]: pgmap v311: 97 pgs: 97 active+clean; 322 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.4 MiB/s wr, 735 op/s 2024-02-10T06:46:51.402 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:51 smithi029 ceph-mon[27077]: from='client.14696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:46:51.570 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:51 smithi008 ceph-mon[24646]: pgmap v311: 97 pgs: 97 active+clean; 322 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.4 MiB/s wr, 735 op/s 2024-02-10T06:46:51.570 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:51 smithi008 ceph-mon[24646]: from='client.14696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:46:51.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 86243 13.90 MB/sec execute 12 sec latency 28.614 ms 2024-02-10T06:46:51.707 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:46:51.707 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (7m) 2m ago 8m 24.7M - ba2b418f427c d582fd72ee55 2024-02-10T06:46:51.707 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (8m) 2m ago 8m 6405k - 17.2.0 e1d6a67b021e 5fa2e5bdd7cc 2024-02-10T06:46:51.708 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (8m) 2m ago 8m 6405k - 17.2.0 e1d6a67b021e 7527ea77b5cc 2024-02-10T06:46:51.708 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (7m) 2m ago 8m 54.5M - 8.3.5 dad864ee21e9 c9887c4d30ec 2024-02-10T06:46:51.708 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (2m) 2m ago 2m 15.0M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:46:51.708 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (2m) 2m ago 2m 12.0M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:46:51.708 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:9283 running (10m) 2m ago 10m 460M - 17.2.0 e1d6a67b021e b0d0b31e96f4 2024-02-10T06:46:51.708 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283 starting - - - - 2024-02-10T06:46:51.708 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (10m) 2m ago 10m 45.6M 2048M 17.2.0 e1d6a67b021e 061312907e4b 2024-02-10T06:46:51.708 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (8m) 2m ago 8m 39.9M 2048M 17.2.0 e1d6a67b021e a790a61584bf 2024-02-10T06:46:51.708 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (2m) 2m ago 2m 51.0M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:46:51.708 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (2m) 2m ago 2m 46.7M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:46:51.708 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (8m) 2m ago 8m 17.3M - 1dbe0e931976 3c06441739f8 2024-02-10T06:46:51.709 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (7m) 2m ago 7m 17.0M - 1dbe0e931976 6e5039760c8a 2024-02-10T06:46:51.709 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (7m) 2m ago 7m 45.5M 3217M 17.2.0 e1d6a67b021e 139b7a92bfc4 2024-02-10T06:46:51.709 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (6m) 2m ago 6m 43.5M 3217M 17.2.0 e1d6a67b021e 0f1119e17a52 2024-02-10T06:46:51.709 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (6m) 2m ago 6m 43.9M 3217M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:46:51.709 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (6m) 2m ago 5m 41.5M 3217M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:46:51.709 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (5m) 2m ago 5m 41.9M 3985M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:46:51.709 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (5m) 2m ago 5m 41.2M 3985M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:46:51.709 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (4m) 2m ago 4m 38.6M 3985M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:46:51.709 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (4m) 2m ago 4m 39.6M 3985M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:46:51.709 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (7m) 2m ago 7m 63.2M - 514e6a882f6e eedfd13fcab7 2024-02-10T06:46:52.327 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:52 smithi008 ceph-mon[24646]: from='client.14700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:46:52.327 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:52 smithi008 ceph-mon[24646]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:46:52.359 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:46:52.359 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:46:52.359 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:46:52.359 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:46:52.359 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:46:52.359 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:46:52.360 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:46:52.360 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:46:52.360 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:46:52.360 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:46:52.360 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:46:52.360 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:46:52.360 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:46:52.360 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:46:52.361 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-02-10T06:46:52.361 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:46:52.361 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:46:52.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 86748 13.68 MB/sec execute 13 sec latency 61.118 ms 2024-02-10T06:46:52.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:52 smithi029 ceph-mon[27077]: from='client.14700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:46:52.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:52 smithi029 ceph-mon[27077]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:46:52.903 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:46:52.903 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:46:52.903 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:46:52.904 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-02-10T06:46:52.904 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "", 2024-02-10T06:46:52.904 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading mgr daemons" 2024-02-10T06:46:52.904 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:46:53.532 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:46:53.591 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:53 smithi008 ceph-mon[24646]: pgmap v312: 97 pgs: 97 active+clean; 318 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 7.6 MiB/s wr, 760 op/s 2024-02-10T06:46:53.591 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:53 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1025526435' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:46:53.591 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:53 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/crt"}]: dispatch 2024-02-10T06:46:53.591 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:53 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:46:53.591 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:53 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/key"}]: dispatch 2024-02-10T06:46:53.591 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:53 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:46:53.592 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:53 smithi008 ceph-mon[24646]: Standby manager daemon smithi029.fuvrtf restarted 2024-02-10T06:46:53.592 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:53 smithi008 ceph-mon[24646]: Standby manager daemon smithi029.fuvrtf started 2024-02-10T06:46:53.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 87277 13.57 MB/sec execute 14 sec latency 75.192 ms 2024-02-10T06:46:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:53 smithi029 ceph-mon[27077]: pgmap v312: 97 pgs: 97 active+clean; 318 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 7.6 MiB/s wr, 760 op/s 2024-02-10T06:46:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:53 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1025526435' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:46:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:53 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/crt"}]: dispatch 2024-02-10T06:46:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:53 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:46:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:53 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/key"}]: dispatch 2024-02-10T06:46:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:53 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:46:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:53 smithi029 ceph-mon[27077]: Standby manager daemon smithi029.fuvrtf restarted 2024-02-10T06:46:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:53 smithi029 ceph-mon[27077]: Standby manager daemon smithi029.fuvrtf started 2024-02-10T06:46:54.614 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:54 smithi008 ceph-mon[24646]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:46:54.614 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:54 smithi008 ceph-mon[24646]: mgrmap e22: smithi008.pxifbt(active, since 8m), standbys: smithi029.fuvrtf 2024-02-10T06:46:54.614 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:54 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1271629584' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:46:54.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 87740 13.64 MB/sec execute 15 sec latency 49.445 ms 2024-02-10T06:46:54.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:54 smithi029 ceph-mon[27077]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:46:54.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:54 smithi029 ceph-mon[27077]: mgrmap e22: smithi008.pxifbt(active, since 8m), standbys: smithi029.fuvrtf 2024-02-10T06:46:54.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:54 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1271629584' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:46:55.380 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:55 smithi008 ceph-mon[24646]: pgmap v313: 97 pgs: 97 active+clean; 320 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.1 MiB/s wr, 674 op/s 2024-02-10T06:46:55.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 88136 13.56 MB/sec execute 16 sec latency 101.798 ms 2024-02-10T06:46:55.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:55 smithi029 ceph-mon[27077]: pgmap v313: 97 pgs: 97 active+clean; 320 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.1 MiB/s wr, 674 op/s 2024-02-10T06:46:56.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 88506 13.54 MB/sec execute 17 sec latency 99.430 ms 2024-02-10T06:46:57.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:57 smithi008 ceph-mon[24646]: pgmap v314: 97 pgs: 97 active+clean; 320 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.5 MiB/s wr, 536 op/s 2024-02-10T06:46:57.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 88976 13.70 MB/sec execute 18 sec latency 32.508 ms 2024-02-10T06:46:57.618 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:57 smithi029 ceph-mon[27077]: pgmap v314: 97 pgs: 97 active+clean; 320 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.5 MiB/s wr, 536 op/s 2024-02-10T06:46:58.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 89371 13.41 MB/sec execute 19 sec latency 107.436 ms 2024-02-10T06:46:59.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:46:59 smithi008 ceph-mon[24646]: pgmap v315: 97 pgs: 97 active+clean; 322 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 768 op/s 2024-02-10T06:46:59.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 89827 13.17 MB/sec execute 20 sec latency 24.864 ms 2024-02-10T06:46:59.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:46:59 smithi029 ceph-mon[27077]: pgmap v315: 97 pgs: 97 active+clean; 322 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 768 op/s 2024-02-10T06:47:00.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 90354 13.05 MB/sec execute 21 sec latency 29.546 ms 2024-02-10T06:47:01.114 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:01 smithi008 ceph-mon[24646]: pgmap v316: 97 pgs: 97 active+clean; 328 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 7.1 MiB/s wr, 662 op/s 2024-02-10T06:47:01.114 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:01 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:47:01.114 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:01 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:47:01.363 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:01 smithi029 ceph-mon[27077]: pgmap v316: 97 pgs: 97 active+clean; 328 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 7.1 MiB/s wr, 662 op/s 2024-02-10T06:47:01.364 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:01 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:47:01.364 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:01 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:47:01.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 90866 12.99 MB/sec execute 22 sec latency 45.515 ms 2024-02-10T06:47:02.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 91256 12.97 MB/sec execute 23 sec latency 163.562 ms 2024-02-10T06:47:03.300 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:03 smithi008 ceph-mon[24646]: pgmap v317: 97 pgs: 97 active+clean; 325 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.5 MiB/s wr, 719 op/s 2024-02-10T06:47:03.440 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:03 smithi029 ceph-mon[27077]: pgmap v317: 97 pgs: 97 active+clean; 325 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.5 MiB/s wr, 719 op/s 2024-02-10T06:47:03.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 91684 13.03 MB/sec execute 24 sec latency 40.098 ms 2024-02-10T06:47:04.617 INFO:teuthology.orchestra.run.smithi008.stdout: 5 92002 12.99 MB/sec execute 25 sec latency 108.656 ms 2024-02-10T06:47:04.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:04 smithi008 ceph-mon[24646]: pgmap v318: 97 pgs: 97 active+clean; 320 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 7.3 MiB/s wr, 705 op/s 2024-02-10T06:47:05.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:04 smithi029 ceph-mon[27077]: pgmap v318: 97 pgs: 97 active+clean; 320 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 7.3 MiB/s wr, 705 op/s 2024-02-10T06:47:05.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 92596 13.24 MB/sec execute 26 sec latency 153.043 ms 2024-02-10T06:47:06.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 93132 13.20 MB/sec execute 27 sec latency 20.426 ms 2024-02-10T06:47:06.922 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:06 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:47:06.922 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:06 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:47:06.922 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:06 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:47:06.922 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:06 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:47:06.922 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:06 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr fail", "who": "smithi008.pxifbt"}]: dispatch 2024-02-10T06:47:06.922 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:06 smithi008 ceph-mon[24646]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T06:47:07.104 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:06 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:47:07.104 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:06 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:47:07.104 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:06 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' 2024-02-10T06:47:07.104 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:06 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:47:07.104 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:06 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr fail", "who": "smithi008.pxifbt"}]: dispatch 2024-02-10T06:47:07.104 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:06 smithi029 ceph-mon[27077]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T06:47:07.617 INFO:teuthology.orchestra.run.smithi008.stdout: 5 93611 12.97 MB/sec execute 28 sec latency 34.043 ms 2024-02-10T06:47:07.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: Upgrade: Need to upgrade myself (mgr.smithi008.pxifbt) 2024-02-10T06:47:07.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: Failing over to other MGR 2024-02-10T06:47:07.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: pgmap v320: 97 pgs: 97 active+clean; 320 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.6 MiB/s wr, 657 op/s 2024-02-10T06:47:07.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: Standby manager daemon smithi008.pxifbt started 2024-02-10T06:47:07.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:47:07.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:47:07.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.vfwcte"}]: dispatch 2024-02-10T06:47:07.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi029.bwmcyt"}]: dispatch 2024-02-10T06:47:07.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mgr metadata", "who": "smithi029.fuvrtf", "id": "smithi029.fuvrtf"}]: dispatch 2024-02-10T06:47:07.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mgr metadata", "who": "smithi008.pxifbt", "id": "smithi008.pxifbt"}]: dispatch 2024-02-10T06:47:07.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:47:07.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:47:07.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:47:07.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:47:07.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:47:07.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:47:07.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:47:07.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:47:07.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:47:07.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:47:07.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:47:07.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "mgr fail", "who": "smithi008.pxifbt"}]': finished 2024-02-10T06:47:07.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: mgrmap e23: smithi029.fuvrtf(active, starting, since 1.00004s), standbys: smithi008.pxifbt 2024-02-10T06:47:07.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: Manager daemon smithi029.fuvrtf is now available 2024-02-10T06:47:07.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:07.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:47:07.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:47:07.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:47:07.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.fuvrtf/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:47:07.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.fuvrtf/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:47:07.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.fuvrtf/trash_purge_schedule"}]: dispatch 2024-02-10T06:47:07.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:07 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.fuvrtf/trash_purge_schedule"}]: dispatch 2024-02-10T06:47:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: Upgrade: Need to upgrade myself (mgr.smithi008.pxifbt) 2024-02-10T06:47:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: Failing over to other MGR 2024-02-10T06:47:08.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: pgmap v320: 97 pgs: 97 active+clean; 320 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.6 MiB/s wr, 657 op/s 2024-02-10T06:47:08.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: Standby manager daemon smithi008.pxifbt started 2024-02-10T06:47:08.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:47:08.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:47:08.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.vfwcte"}]: dispatch 2024-02-10T06:47:08.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi029.bwmcyt"}]: dispatch 2024-02-10T06:47:08.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mgr metadata", "who": "smithi029.fuvrtf", "id": "smithi029.fuvrtf"}]: dispatch 2024-02-10T06:47:08.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mgr metadata", "who": "smithi008.pxifbt", "id": "smithi008.pxifbt"}]: dispatch 2024-02-10T06:47:08.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:47:08.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:47:08.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:47:08.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:47:08.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:47:08.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:47:08.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:47:08.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:47:08.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:47:08.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:47:08.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:47:08.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.14202 172.21.15.8:0/541602001' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "mgr fail", "who": "smithi008.pxifbt"}]': finished 2024-02-10T06:47:08.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: mgrmap e23: smithi029.fuvrtf(active, starting, since 1.00004s), standbys: smithi008.pxifbt 2024-02-10T06:47:08.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: Manager daemon smithi029.fuvrtf is now available 2024-02-10T06:47:08.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:08.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:47:08.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:47:08.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:47:08.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.fuvrtf/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:47:08.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.fuvrtf/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:47:08.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.fuvrtf/trash_purge_schedule"}]: dispatch 2024-02-10T06:47:08.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:07 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.fuvrtf/trash_purge_schedule"}]: dispatch 2024-02-10T06:47:08.616 INFO:teuthology.orchestra.run.smithi008.stdout: 5 94134 12.96 MB/sec execute 29 sec latency 36.186 ms 2024-02-10T06:47:08.939 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:08 smithi008 ceph-mon[24646]: Deploying cephadm binary to smithi029 2024-02-10T06:47:08.939 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:08 smithi008 ceph-mon[24646]: Deploying cephadm binary to smithi008 2024-02-10T06:47:08.939 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:08 smithi008 ceph-mon[24646]: mgrmap e24: smithi029.fuvrtf(active, since 2s), standbys: smithi008.pxifbt 2024-02-10T06:47:08.939 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:08 smithi008 ceph-mon[24646]: pgmap v3: 97 pgs: 97 active+clean; 321 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-02-10T06:47:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:08 smithi029 ceph-mon[27077]: Deploying cephadm binary to smithi029 2024-02-10T06:47:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:08 smithi029 ceph-mon[27077]: Deploying cephadm binary to smithi008 2024-02-10T06:47:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:08 smithi029 ceph-mon[27077]: mgrmap e24: smithi029.fuvrtf(active, since 2s), standbys: smithi008.pxifbt 2024-02-10T06:47:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:08 smithi029 ceph-mon[27077]: pgmap v3: 97 pgs: 97 active+clean; 321 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-02-10T06:47:09.617 INFO:teuthology.orchestra.run.smithi008.stdout: 5 94703 12.99 MB/sec execute 30 sec latency 39.416 ms 2024-02-10T06:47:09.965 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:09 smithi008 ceph-mon[24646]: pgmap v4: 97 pgs: 97 active+clean; 321 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-02-10T06:47:09.965 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:09 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:09.965 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:09 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:10.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:09 smithi029 ceph-mon[27077]: pgmap v4: 97 pgs: 97 active+clean; 321 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-02-10T06:47:10.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:09 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:10.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:09 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:10.617 INFO:teuthology.orchestra.run.smithi008.stdout: 5 95465 13.52 MB/sec execute 31 sec latency 23.994 ms 2024-02-10T06:47:10.975 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:10 smithi008 ceph-mon[24646]: [10/Feb/2024:06:47:09] ENGINE Bus STARTING 2024-02-10T06:47:10.975 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:10 smithi008 ceph-mon[24646]: mgrmap e25: smithi029.fuvrtf(active, since 4s), standbys: smithi008.pxifbt 2024-02-10T06:47:11.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:10 smithi029 ceph-mon[27077]: [10/Feb/2024:06:47:09] ENGINE Bus STARTING 2024-02-10T06:47:11.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:10 smithi029 ceph-mon[27077]: mgrmap e25: smithi029.fuvrtf(active, since 4s), standbys: smithi008.pxifbt 2024-02-10T06:47:11.617 INFO:teuthology.orchestra.run.smithi008.stdout: 5 95763 13.35 MB/sec execute 32 sec latency 53.982 ms 2024-02-10T06:47:11.992 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:11 smithi008 ceph-mon[24646]: [10/Feb/2024:06:47:10] ENGINE Serving on https://172.21.15.29:7150 2024-02-10T06:47:11.992 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:11 smithi008 ceph-mon[24646]: [10/Feb/2024:06:47:10] ENGINE Serving on http://172.21.15.29:8765 2024-02-10T06:47:11.992 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:11 smithi008 ceph-mon[24646]: [10/Feb/2024:06:47:10] ENGINE Bus STARTED 2024-02-10T06:47:11.992 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:11 smithi008 ceph-mon[24646]: pgmap v5: 97 pgs: 97 active+clean; 321 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-02-10T06:47:12.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:11 smithi029 ceph-mon[27077]: [10/Feb/2024:06:47:10] ENGINE Serving on https://172.21.15.29:7150 2024-02-10T06:47:12.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:11 smithi029 ceph-mon[27077]: [10/Feb/2024:06:47:10] ENGINE Serving on http://172.21.15.29:8765 2024-02-10T06:47:12.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:11 smithi029 ceph-mon[27077]: [10/Feb/2024:06:47:10] ENGINE Bus STARTED 2024-02-10T06:47:12.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:11 smithi029 ceph-mon[27077]: pgmap v5: 97 pgs: 97 active+clean; 321 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-02-10T06:47:12.617 INFO:teuthology.orchestra.run.smithi008.stdout: 5 96101 13.28 MB/sec execute 33 sec latency 39.241 ms 2024-02-10T06:47:13.617 INFO:teuthology.orchestra.run.smithi008.stdout: 5 96363 13.09 MB/sec execute 34 sec latency 61.843 ms 2024-02-10T06:47:14.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:13 smithi008 ceph-mon[24646]: pgmap v6: 97 pgs: 97 active+clean; 321 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-02-10T06:47:14.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:13 smithi029 ceph-mon[27077]: pgmap v6: 97 pgs: 97 active+clean; 321 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-02-10T06:47:14.617 INFO:teuthology.orchestra.run.smithi008.stdout: 5 96734 12.93 MB/sec execute 35 sec latency 41.173 ms 2024-02-10T06:47:15.617 INFO:teuthology.orchestra.run.smithi008.stdout: 5 97208 12.86 MB/sec execute 36 sec latency 29.126 ms 2024-02-10T06:47:16.064 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:15 smithi008 ceph-mon[24646]: pgmap v7: 97 pgs: 97 active+clean; 323 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 5.3 MiB/s wr, 492 op/s 2024-02-10T06:47:16.064 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:15 smithi008 ceph-mon[24646]: Standby manager daemon smithi008.pxifbt restarted 2024-02-10T06:47:16.064 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:15 smithi008 ceph-mon[24646]: Standby manager daemon smithi008.pxifbt started 2024-02-10T06:47:16.065 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:15 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.8:0/1619786174' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.pxifbt/crt"}]: dispatch 2024-02-10T06:47:16.065 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:15 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.8:0/1619786174' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:47:16.065 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:15 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.8:0/1619786174' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.pxifbt/key"}]: dispatch 2024-02-10T06:47:16.065 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:15 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.8:0/1619786174' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:47:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:15 smithi029 ceph-mon[27077]: pgmap v7: 97 pgs: 97 active+clean; 323 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 5.3 MiB/s wr, 492 op/s 2024-02-10T06:47:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:15 smithi029 ceph-mon[27077]: Standby manager daemon smithi008.pxifbt restarted 2024-02-10T06:47:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:15 smithi029 ceph-mon[27077]: Standby manager daemon smithi008.pxifbt started 2024-02-10T06:47:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:15 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.8:0/1619786174' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.pxifbt/crt"}]: dispatch 2024-02-10T06:47:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:15 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.8:0/1619786174' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:47:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:15 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.8:0/1619786174' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.pxifbt/key"}]: dispatch 2024-02-10T06:47:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:15 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.8:0/1619786174' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:47:16.617 INFO:teuthology.orchestra.run.smithi008.stdout: 5 97856 12.88 MB/sec execute 37 sec latency 26.762 ms 2024-02-10T06:47:16.938 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:16 smithi008 ceph-mon[24646]: mgrmap e26: smithi029.fuvrtf(active, since 10s), standbys: smithi008.pxifbt 2024-02-10T06:47:17.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:16 smithi029 ceph-mon[27077]: mgrmap e26: smithi029.fuvrtf(active, since 10s), standbys: smithi008.pxifbt 2024-02-10T06:47:17.617 INFO:teuthology.orchestra.run.smithi008.stdout: 5 98402 13.00 MB/sec execute 38 sec latency 68.023 ms 2024-02-10T06:47:18.095 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:17 smithi008 ceph-mon[24646]: pgmap v8: 97 pgs: 97 active+clean; 323 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 4.1 MiB/s wr, 382 op/s 2024-02-10T06:47:18.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:17 smithi029 ceph-mon[27077]: pgmap v8: 97 pgs: 97 active+clean; 323 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 4.1 MiB/s wr, 382 op/s 2024-02-10T06:47:18.617 INFO:teuthology.orchestra.run.smithi008.stdout: 5 99016 13.18 MB/sec execute 39 sec latency 20.371 ms 2024-02-10T06:47:19.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:19 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:19.102 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:19 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:19.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:19 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:19.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:19 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:19.617 INFO:teuthology.orchestra.run.smithi008.stdout: 5 99415 13.18 MB/sec execute 40 sec latency 48.308 ms 2024-02-10T06:47:20.116 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:20 smithi008 ceph-mon[24646]: pgmap v9: 97 pgs: 97 active+clean; 317 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 6.3 MiB/s wr, 635 op/s 2024-02-10T06:47:20.116 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:20 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:20.116 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:20 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:20.246 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:20 smithi029 ceph-mon[27077]: pgmap v9: 97 pgs: 97 active+clean; 317 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 6.3 MiB/s wr, 635 op/s 2024-02-10T06:47:20.246 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:20 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:20.246 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:20 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:20.617 INFO:teuthology.orchestra.run.smithi008.stdout: 5 99784 13.10 MB/sec execute 41 sec latency 31.803 ms 2024-02-10T06:47:21.618 INFO:teuthology.orchestra.run.smithi008.stdout: 5 100140 13.00 MB/sec execute 42 sec latency 39.525 ms 2024-02-10T06:47:22.139 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:22 smithi008 ceph-mon[24646]: pgmap v10: 97 pgs: 97 active+clean; 317 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 5.8 MiB/s wr, 580 op/s 2024-02-10T06:47:22.139 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:22 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:47:22.388 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:22 smithi029 ceph-mon[27077]: pgmap v10: 97 pgs: 97 active+clean; 317 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 5.8 MiB/s wr, 580 op/s 2024-02-10T06:47:22.388 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:22 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:47:22.617 INFO:teuthology.orchestra.run.smithi008.stdout: 5 100474 12.80 MB/sec execute 43 sec latency 35.873 ms 2024-02-10T06:47:23.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:23 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:23.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:23 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:23.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:23 smithi008 ceph-mon[24646]: pgmap v11: 97 pgs: 97 active+clean; 317 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 5.8 MiB/s wr, 580 op/s 2024-02-10T06:47:23.403 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:23 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:23.403 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:23 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:23.403 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:23 smithi029 ceph-mon[27077]: pgmap v11: 97 pgs: 97 active+clean; 317 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 5.8 MiB/s wr, 580 op/s 2024-02-10T06:47:23.618 INFO:teuthology.orchestra.run.smithi008.stdout: 5 100833 12.72 MB/sec execute 44 sec latency 57.929 ms 2024-02-10T06:47:24.049 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:47:24.491 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:24.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:24.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: Adjusting osd_memory_target on smithi029 to 2705M 2024-02-10T06:47:24.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:24.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:24.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:24.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: Adjusting osd_memory_target on smithi008 to 1937M 2024-02-10T06:47:24.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:24.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:47:24.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:47:24.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: Updating smithi008:/etc/ceph/ceph.conf 2024-02-10T06:47:24.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: Updating smithi029:/etc/ceph/ceph.conf 2024-02-10T06:47:24.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:47:24.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: Updating smithi029:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:47:24.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:24 smithi008 ceph-mon[24646]: Updating smithi008:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:47:24.618 INFO:teuthology.orchestra.run.smithi008.stdout: 5 101302 12.67 MB/sec execute 45 sec latency 40.951 ms 2024-02-10T06:47:24.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:24.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:24.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: Adjusting osd_memory_target on smithi029 to 2705M 2024-02-10T06:47:24.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:24.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:24.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:24.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:47:24.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: Adjusting osd_memory_target on smithi008 to 1937M 2024-02-10T06:47:24.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:24.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:47:24.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:47:24.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: Updating smithi008:/etc/ceph/ceph.conf 2024-02-10T06:47:24.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: Updating smithi029:/etc/ceph/ceph.conf 2024-02-10T06:47:24.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:47:24.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: Updating smithi029:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:47:24.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:24 smithi029 ceph-mon[27077]: Updating smithi008:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:47:24.964 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:47:24.964 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (7m) 2s ago 9m 28.0M - 0.23.0 ba2b418f427c d582fd72ee55 2024-02-10T06:47:24.964 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (9m) 2s ago 9m 6405k - 17.2.0 e1d6a67b021e 5fa2e5bdd7cc 2024-02-10T06:47:24.964 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (8m) 6s ago 8m 6405k - 17.2.0 e1d6a67b021e 7527ea77b5cc 2024-02-10T06:47:24.964 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (7m) 2s ago 9m 54.9M - 8.3.5 dad864ee21e9 c9887c4d30ec 2024-02-10T06:47:24.964 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (3m) 2s ago 3m 36.7M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:47:24.964 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (3m) 6s ago 3m 13.0M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:47:24.964 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:9283 running (10m) 2s ago 10m 355M - 17.2.0 e1d6a67b021e b0d0b31e96f4 2024-02-10T06:47:24.964 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283 running (40s) 6s ago 8m 490M - 19.0.0-1361-g290212a1 c328ab9dcfc7 754f1049995b 2024-02-10T06:47:24.964 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (10m) 2s ago 10m 52.0M 2048M 17.2.0 e1d6a67b021e 061312907e4b 2024-02-10T06:47:24.965 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (8m) 6s ago 8m 41.7M 2048M 17.2.0 e1d6a67b021e a790a61584bf 2024-02-10T06:47:24.965 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (2m) 2s ago 2m 249M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:47:24.965 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (2m) 6s ago 2m 73.8M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:47:24.965 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (9m) 2s ago 9m 20.0M - 1.3.1 1dbe0e931976 3c06441739f8 2024-02-10T06:47:24.965 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (8m) 6s ago 8m 19.5M - 1.3.1 1dbe0e931976 6e5039760c8a 2024-02-10T06:47:24.965 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (7m) 2s ago 7m 365M 1937M 17.2.0 e1d6a67b021e 139b7a92bfc4 2024-02-10T06:47:24.965 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (7m) 2s ago 7m 272M 1937M 17.2.0 e1d6a67b021e 0f1119e17a52 2024-02-10T06:47:24.965 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (7m) 2s ago 7m 252M 1937M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:47:24.965 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (6m) 2s ago 6m 336M 1937M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:47:24.965 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (6m) 6s ago 6m 345M 2705M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:47:24.965 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (5m) 6s ago 5m 372M 2705M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:47:24.965 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (5m) 6s ago 5m 306M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:47:24.965 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (4m) 6s ago 4m 459M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:47:24.965 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (8m) 2s ago 8m 67.5M - 2.33.4 514e6a882f6e eedfd13fcab7 2024-02-10T06:47:25.509 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:47:25.533 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:47:25.533 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:47:25.533 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:47:25.533 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:47:25.533 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T06:47:25.534 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T06:47:25.534 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:47:25.534 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:47:25.534 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:47:25.534 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:47:25.534 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:47:25.534 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:47:25.534 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:47:25.534 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:47:25.534 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-10T06:47:25.534 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T06:47:25.534 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:47:25.534 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:47:25.565 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:25 smithi008 ceph-mon[24646]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:47:25.565 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:25 smithi008 ceph-mon[24646]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:47:25.565 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:25 smithi008 ceph-mon[24646]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:47:25.565 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:25 smithi008 ceph-mon[24646]: Updating smithi029:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:47:25.565 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:25 smithi008 ceph-mon[24646]: Updating smithi008:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:47:25.565 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:25 smithi008 ceph-mon[24646]: pgmap v12: 97 pgs: 97 active+clean; 321 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 8.4 MiB/s wr, 830 op/s 2024-02-10T06:47:25.565 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:25 smithi008 ceph-mon[24646]: from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:47:25.565 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:25 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:25.565 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:25 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:25.565 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:25 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:25.566 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:25 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:25.566 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:25 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:25.566 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:25 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:25.566 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:25 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:25.618 INFO:teuthology.orchestra.run.smithi008.stdout: 5 101504 12.49 MB/sec execute 46 sec latency 331.676 ms 2024-02-10T06:47:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:25 smithi029 ceph-mon[27077]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:47:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:25 smithi029 ceph-mon[27077]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:47:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:25 smithi029 ceph-mon[27077]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:47:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:25 smithi029 ceph-mon[27077]: Updating smithi029:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:47:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:25 smithi029 ceph-mon[27077]: Updating smithi008:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:47:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:25 smithi029 ceph-mon[27077]: pgmap v12: 97 pgs: 97 active+clean; 321 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 8.4 MiB/s wr, 830 op/s 2024-02-10T06:47:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:25 smithi029 ceph-mon[27077]: from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:47:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:25 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:25 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:25 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:25 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:25 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:25 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:25.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:25 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:26.060 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:47:26.060 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:47:26.060 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:47:26.061 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:47:26.061 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-02-10T06:47:26.061 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "1/23 daemons upgraded", 2024-02-10T06:47:26.061 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-02-10T06:47:26.061 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:47:26.061 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:47:26.618 INFO:teuthology.orchestra.run.smithi008.stdout: 5 101930 12.53 MB/sec execute 47 sec latency 127.469 ms 2024-02-10T06:47:26.632 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:47:26.685 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:26 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:26.685 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:26 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:26.685 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:26 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:26.685 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:26 smithi008 ceph-mon[24646]: Reconfiguring alertmanager.smithi008 (dependencies changed)... 2024-02-10T06:47:26.685 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:26 smithi008 ceph-mon[24646]: Reconfiguring daemon alertmanager.smithi008 on smithi008 2024-02-10T06:47:26.685 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:26 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/22428392' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:47:26.685 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:26 smithi008 ceph-mon[24646]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:47:26.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:26 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:26.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:26 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:26.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:26 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:26.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:26 smithi029 ceph-mon[27077]: Reconfiguring alertmanager.smithi008 (dependencies changed)... 2024-02-10T06:47:26.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:26 smithi029 ceph-mon[27077]: Reconfiguring daemon alertmanager.smithi008 on smithi008 2024-02-10T06:47:26.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:26 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/22428392' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:47:26.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:26 smithi029 ceph-mon[27077]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:47:27.618 INFO:teuthology.orchestra.run.smithi008.stdout: 5 102342 12.51 MB/sec execute 48 sec latency 38.913 ms 2024-02-10T06:47:27.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:27 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1138254629' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:47:27.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:27 smithi008 ceph-mon[24646]: pgmap v13: 97 pgs: 97 active+clean; 321 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.3 MiB/s wr, 544 op/s 2024-02-10T06:47:27.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:27 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1138254629' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:47:27.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:27 smithi029 ceph-mon[27077]: pgmap v13: 97 pgs: 97 active+clean; 321 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.3 MiB/s wr, 544 op/s 2024-02-10T06:47:28.618 INFO:teuthology.orchestra.run.smithi008.stdout: 5 102727 12.55 MB/sec execute 49 sec latency 41.516 ms 2024-02-10T06:47:29.618 INFO:teuthology.orchestra.run.smithi008.stdout: 5 103083 12.55 MB/sec execute 50 sec latency 32.503 ms 2024-02-10T06:47:30.018 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:29 smithi008 ceph-mon[24646]: pgmap v14: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 765 op/s 2024-02-10T06:47:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:29 smithi029 ceph-mon[27077]: pgmap v14: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 765 op/s 2024-02-10T06:47:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: 5 103365 12.41 MB/sec execute 51 sec latency 51.669 ms 2024-02-10T06:47:31.618 INFO:teuthology.orchestra.run.smithi008.stdout: 5 103789 12.37 MB/sec execute 52 sec latency 30.740 ms 2024-02-10T06:47:32.041 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:31 smithi008 ceph-mon[24646]: pgmap v15: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.9 MiB/s wr, 470 op/s 2024-02-10T06:47:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:31 smithi029 ceph-mon[27077]: pgmap v15: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.9 MiB/s wr, 470 op/s 2024-02-10T06:47:32.618 INFO:teuthology.orchestra.run.smithi008.stdout: 5 104352 12.36 MB/sec execute 53 sec latency 51.172 ms 2024-02-10T06:47:33.618 INFO:teuthology.orchestra.run.smithi008.stdout: 5 104952 12.37 MB/sec execute 54 sec latency 25.658 ms 2024-02-10T06:47:34.049 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:33 smithi008 ceph-mon[24646]: pgmap v16: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.9 MiB/s wr, 470 op/s 2024-02-10T06:47:34.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:33 smithi029 ceph-mon[27077]: pgmap v16: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.9 MiB/s wr, 470 op/s 2024-02-10T06:47:34.618 INFO:teuthology.orchestra.run.smithi008.stdout: 5 105511 12.45 MB/sec execute 55 sec latency 39.317 ms 2024-02-10T06:47:35.618 INFO:teuthology.orchestra.run.smithi008.stdout: 5 106117 12.60 MB/sec execute 56 sec latency 22.553 ms 2024-02-10T06:47:36.062 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:35 smithi008 ceph-mon[24646]: pgmap v17: 97 pgs: 97 active+clean; 323 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 723 op/s 2024-02-10T06:47:36.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:35 smithi029 ceph-mon[27077]: pgmap v17: 97 pgs: 97 active+clean; 323 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 723 op/s 2024-02-10T06:47:36.624 INFO:teuthology.orchestra.run.smithi008.stdout: 5 106586 12.66 MB/sec execute 57 sec latency 36.943 ms 2024-02-10T06:47:37.625 INFO:teuthology.orchestra.run.smithi008.stdout: 5 106911 12.56 MB/sec execute 58 sec latency 37.360 ms 2024-02-10T06:47:38.088 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:37 smithi008 ceph-mon[24646]: pgmap v18: 97 pgs: 97 active+clean; 323 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.7 MiB/s wr, 474 op/s 2024-02-10T06:47:38.088 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:37 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:47:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:37 smithi029 ceph-mon[27077]: pgmap v18: 97 pgs: 97 active+clean; 323 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.7 MiB/s wr, 474 op/s 2024-02-10T06:47:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:37 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:47:38.625 INFO:teuthology.orchestra.run.smithi008.stdout: 5 107263 12.50 MB/sec execute 59 sec latency 37.536 ms 2024-02-10T06:47:39.625 INFO:teuthology.orchestra.run.smithi008.stdout: 5 107615 12.37 MB/sec execute 60 sec latency 48.699 ms 2024-02-10T06:47:40.117 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:39 smithi008 ceph-mon[24646]: pgmap v19: 97 pgs: 97 active+clean; 328 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 8.1 MiB/s wr, 770 op/s 2024-02-10T06:47:40.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:39 smithi029 ceph-mon[27077]: pgmap v19: 97 pgs: 97 active+clean; 328 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 8.1 MiB/s wr, 770 op/s 2024-02-10T06:47:40.625 INFO:teuthology.orchestra.run.smithi008.stdout: 5 107947 12.31 MB/sec execute 61 sec latency 101.447 ms 2024-02-10T06:47:41.625 INFO:teuthology.orchestra.run.smithi008.stdout: 5 108409 12.28 MB/sec execute 62 sec latency 47.195 ms 2024-02-10T06:47:42.155 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:41 smithi008 ceph-mon[24646]: pgmap v20: 97 pgs: 97 active+clean; 328 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.8 MiB/s wr, 549 op/s 2024-02-10T06:47:42.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:41 smithi029 ceph-mon[27077]: pgmap v20: 97 pgs: 97 active+clean; 328 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.8 MiB/s wr, 549 op/s 2024-02-10T06:47:42.625 INFO:teuthology.orchestra.run.smithi008.stdout: 5 108795 12.21 MB/sec execute 63 sec latency 42.062 ms 2024-02-10T06:47:43.625 INFO:teuthology.orchestra.run.smithi008.stdout: 5 109416 12.37 MB/sec execute 64 sec latency 36.402 ms 2024-02-10T06:47:43.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:43 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:43.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:43 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:43.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:43 smithi008 ceph-mon[24646]: Reconfiguring grafana.smithi008 (dependencies changed)... 2024-02-10T06:47:43.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:43 smithi008 ceph-mon[24646]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T06:47:43.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:43 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:43.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:43 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:43.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:43 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:47:43.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:43 smithi008 ceph-mon[24646]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:47:43.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:43 smithi008 ceph-mon[24646]: Reconfiguring daemon grafana.smithi008 on smithi008 2024-02-10T06:47:43.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:43 smithi008 ceph-mon[24646]: pgmap v21: 97 pgs: 97 active+clean; 328 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.8 MiB/s wr, 549 op/s 2024-02-10T06:47:43.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:43 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:43.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:43 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:43.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:43 smithi029 ceph-mon[27077]: Reconfiguring grafana.smithi008 (dependencies changed)... 2024-02-10T06:47:43.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:43 smithi029 ceph-mon[27077]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T06:47:43.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:43 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:43.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:43 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:47:43.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:43 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:47:43.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:43 smithi029 ceph-mon[27077]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:47:43.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:43 smithi029 ceph-mon[27077]: Reconfiguring daemon grafana.smithi008 on smithi008 2024-02-10T06:47:43.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:43 smithi029 ceph-mon[27077]: pgmap v21: 97 pgs: 97 active+clean; 328 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.8 MiB/s wr, 549 op/s 2024-02-10T06:47:44.625 INFO:teuthology.orchestra.run.smithi008.stdout: 5 109965 12.47 MB/sec execute 65 sec latency 30.492 ms 2024-02-10T06:47:45.629 INFO:teuthology.orchestra.run.smithi008.stdout: 5 110488 12.49 MB/sec execute 66 sec latency 28.458 ms 2024-02-10T06:47:45.929 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:45 smithi008 ceph-mon[24646]: pgmap v22: 97 pgs: 97 active+clean; 325 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 792 op/s 2024-02-10T06:47:46.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:45 smithi029 ceph-mon[27077]: pgmap v22: 97 pgs: 97 active+clean; 325 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 792 op/s 2024-02-10T06:47:46.629 INFO:teuthology.orchestra.run.smithi008.stdout: 5 111068 12.49 MB/sec execute 67 sec latency 29.835 ms 2024-02-10T06:47:47.629 INFO:teuthology.orchestra.run.smithi008.stdout: 5 111476 12.45 MB/sec execute 68 sec latency 60.987 ms 2024-02-10T06:47:47.954 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:47 smithi008 ceph-mon[24646]: pgmap v23: 97 pgs: 97 active+clean; 325 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.4 MiB/s wr, 539 op/s 2024-02-10T06:47:48.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:47 smithi029 ceph-mon[27077]: pgmap v23: 97 pgs: 97 active+clean; 325 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.4 MiB/s wr, 539 op/s 2024-02-10T06:47:48.629 INFO:teuthology.orchestra.run.smithi008.stdout: 5 112007 12.45 MB/sec execute 69 sec latency 42.674 ms 2024-02-10T06:47:49.629 INFO:teuthology.orchestra.run.smithi008.stdout: 5 112423 12.40 MB/sec execute 70 sec latency 38.931 ms 2024-02-10T06:47:50.043 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:49 smithi008 ceph-mon[24646]: pgmap v24: 97 pgs: 97 active+clean; 322 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 858 op/s 2024-02-10T06:47:50.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:49 smithi029 ceph-mon[27077]: pgmap v24: 97 pgs: 97 active+clean; 322 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 858 op/s 2024-02-10T06:47:50.629 INFO:teuthology.orchestra.run.smithi008.stdout: 5 112807 12.44 MB/sec execute 71 sec latency 50.346 ms 2024-02-10T06:47:51.629 INFO:teuthology.orchestra.run.smithi008.stdout: 5 113243 12.49 MB/sec execute 72 sec latency 35.017 ms 2024-02-10T06:47:52.166 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:51 smithi008 ceph-mon[24646]: pgmap v25: 97 pgs: 97 active+clean; 322 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.4 MiB/s wr, 562 op/s 2024-02-10T06:47:52.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:51 smithi029 ceph-mon[27077]: pgmap v25: 97 pgs: 97 active+clean; 322 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.4 MiB/s wr, 562 op/s 2024-02-10T06:47:52.629 INFO:teuthology.orchestra.run.smithi008.stdout: 5 113508 12.41 MB/sec execute 73 sec latency 112.332 ms 2024-02-10T06:47:52.936 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:52 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:47:53.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:52 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:47:53.629 INFO:teuthology.orchestra.run.smithi008.stdout: 5 113852 12.40 MB/sec execute 74 sec latency 44.490 ms 2024-02-10T06:47:53.952 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:53 smithi008 ceph-mon[24646]: pgmap v26: 97 pgs: 97 active+clean; 322 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.4 MiB/s wr, 562 op/s 2024-02-10T06:47:54.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:53 smithi029 ceph-mon[27077]: pgmap v26: 97 pgs: 97 active+clean; 322 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.4 MiB/s wr, 562 op/s 2024-02-10T06:47:54.629 INFO:teuthology.orchestra.run.smithi008.stdout: 5 114192 12.32 MB/sec execute 75 sec latency 44.956 ms 2024-02-10T06:47:55.629 INFO:teuthology.orchestra.run.smithi008.stdout: 5 114529 12.25 MB/sec execute 76 sec latency 52.489 ms 2024-02-10T06:47:55.991 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:55 smithi008 ceph-mon[24646]: pgmap v27: 97 pgs: 97 active+clean; 331 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 791 op/s 2024-02-10T06:47:56.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:55 smithi029 ceph-mon[27077]: pgmap v27: 97 pgs: 97 active+clean; 331 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 791 op/s 2024-02-10T06:47:56.629 INFO:teuthology.orchestra.run.smithi008.stdout: 5 114921 12.22 MB/sec execute 77 sec latency 42.335 ms 2024-02-10T06:47:57.177 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:47:57.629 INFO:teuthology.orchestra.run.smithi008.stdout: 5 115336 12.15 MB/sec execute 78 sec latency 45.125 ms 2024-02-10T06:47:58.017 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:57 smithi008 ceph-mon[24646]: pgmap v28: 97 pgs: 97 active+clean; 331 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.9 MiB/s wr, 548 op/s 2024-02-10T06:47:58.017 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:57 smithi008 ceph-mon[24646]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:47:58.199 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:47:58.199 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 starting - - - - 2024-02-10T06:47:58.199 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (9m) 35s ago 9m 6405k - 17.2.0 e1d6a67b021e 5fa2e5bdd7cc 2024-02-10T06:47:58.199 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (9m) 40s ago 9m 6405k - 17.2.0 e1d6a67b021e 7527ea77b5cc 2024-02-10T06:47:58.199 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (8m) 35s ago 9m 54.9M - 8.3.5 dad864ee21e9 c9887c4d30ec 2024-02-10T06:47:58.200 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (3m) 35s ago 3m 36.7M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:47:58.200 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (3m) 40s ago 3m 13.0M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:47:58.200 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:9283 running (11m) 35s ago 11m 355M - 17.2.0 e1d6a67b021e b0d0b31e96f4 2024-02-10T06:47:58.200 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283 running (73s) 40s ago 9m 490M - 19.0.0-1361-g290212a1 c328ab9dcfc7 754f1049995b 2024-02-10T06:47:58.200 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (11m) 35s ago 11m 52.0M 2048M 17.2.0 e1d6a67b021e 061312907e4b 2024-02-10T06:47:58.200 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (9m) 40s ago 9m 41.7M 2048M 17.2.0 e1d6a67b021e a790a61584bf 2024-02-10T06:47:58.200 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (3m) 35s ago 3m 249M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:47:58.200 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (3m) 40s ago 3m 73.8M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:47:58.200 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (9m) 35s ago 9m 20.0M - 1.3.1 1dbe0e931976 3c06441739f8 2024-02-10T06:47:58.200 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (9m) 40s ago 9m 19.5M - 1.3.1 1dbe0e931976 6e5039760c8a 2024-02-10T06:47:58.200 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (8m) 35s ago 8m 365M 1937M 17.2.0 e1d6a67b021e 139b7a92bfc4 2024-02-10T06:47:58.200 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (8m) 35s ago 8m 272M 1937M 17.2.0 e1d6a67b021e 0f1119e17a52 2024-02-10T06:47:58.200 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (7m) 35s ago 7m 252M 1937M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:47:58.200 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (7m) 35s ago 7m 336M 1937M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:47:58.201 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (6m) 40s ago 6m 345M 2705M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:47:58.201 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (6m) 40s ago 6m 372M 2705M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:47:58.201 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (5m) 40s ago 5m 306M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:47:58.201 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (5m) 40s ago 5m 459M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:47:58.201 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (8m) 35s ago 8m 67.5M - 2.33.4 514e6a882f6e eedfd13fcab7 2024-02-10T06:47:58.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:57 smithi029 ceph-mon[27077]: pgmap v28: 97 pgs: 97 active+clean; 331 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.9 MiB/s wr, 548 op/s 2024-02-10T06:47:58.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:57 smithi029 ceph-mon[27077]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:47:58.629 INFO:teuthology.orchestra.run.smithi008.stdout: 5 115723 12.12 MB/sec execute 79 sec latency 55.545 ms 2024-02-10T06:47:58.841 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:47:58.841 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:47:58.841 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:47:58.841 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:47:58.842 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:47:58.842 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T06:47:58.842 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T06:47:58.842 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:47:58.842 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:47:58.842 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:47:58.842 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:47:58.842 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:47:58.842 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:47:58.842 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:47:58.842 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:47:58.842 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-10T06:47:58.843 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T06:47:58.843 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:47:58.843 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:47:59.049 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:58 smithi008 ceph-mon[24646]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:47:59.049 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:58 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/1674887027' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:47:59.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:58 smithi029 ceph-mon[27077]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:47:59.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:58 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/1674887027' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:47:59.382 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:47:59.382 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:47:59.382 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:47:59.382 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:47:59.382 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-02-10T06:47:59.382 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "1/23 daemons upgraded", 2024-02-10T06:47:59.382 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-02-10T06:47:59.382 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:47:59.382 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:47:59.630 INFO:teuthology.orchestra.run.smithi008.stdout: 5 116167 12.18 MB/sec execute 80 sec latency 35.752 ms 2024-02-10T06:48:00.033 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:48:00.058 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:59 smithi008 ceph-mon[24646]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:48:00.058 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:47:59 smithi008 ceph-mon[24646]: pgmap v29: 97 pgs: 97 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 774 op/s 2024-02-10T06:48:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:59 smithi029 ceph-mon[27077]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:48:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:47:59 smithi029 ceph-mon[27077]: pgmap v29: 97 pgs: 97 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 774 op/s 2024-02-10T06:48:00.630 INFO:teuthology.orchestra.run.smithi008.stdout: 5 116556 12.14 MB/sec execute 81 sec latency 50.073 ms 2024-02-10T06:48:01.079 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:00 smithi008 ceph-mon[24646]: from='client.24523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:48:01.079 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:00 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/2810063275' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:48:01.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:00 smithi029 ceph-mon[27077]: from='client.24523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:48:01.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:00 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/2810063275' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:48:01.630 INFO:teuthology.orchestra.run.smithi008.stdout: 5 117063 12.21 MB/sec execute 82 sec latency 26.895 ms 2024-02-10T06:48:02.090 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:01 smithi008 ceph-mon[24646]: pgmap v30: 97 pgs: 97 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-02-10T06:48:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:01 smithi029 ceph-mon[27077]: pgmap v30: 97 pgs: 97 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-02-10T06:48:02.630 INFO:teuthology.orchestra.run.smithi008.stdout: 5 117565 12.22 MB/sec execute 83 sec latency 34.277 ms 2024-02-10T06:48:03.630 INFO:teuthology.orchestra.run.smithi008.stdout: 5 118245 12.24 MB/sec execute 84 sec latency 24.168 ms 2024-02-10T06:48:04.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:03 smithi008 ceph-mon[24646]: pgmap v31: 97 pgs: 97 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-02-10T06:48:04.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:03 smithi029 ceph-mon[27077]: pgmap v31: 97 pgs: 97 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-02-10T06:48:04.630 INFO:teuthology.orchestra.run.smithi008.stdout: 5 119010 12.33 MB/sec execute 85 sec latency 30.947 ms 2024-02-10T06:48:05.631 INFO:teuthology.orchestra.run.smithi008.stdout: 5 119704 12.44 MB/sec execute 86 sec latency 33.784 ms 2024-02-10T06:48:06.114 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:05 smithi008 ceph-mon[24646]: pgmap v32: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.9 KiB/s rd, 7.7 MiB/s wr, 723 op/s 2024-02-10T06:48:06.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:05 smithi029 ceph-mon[27077]: pgmap v32: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.9 KiB/s rd, 7.7 MiB/s wr, 723 op/s 2024-02-10T06:48:06.630 INFO:teuthology.orchestra.run.smithi008.stdout: 5 120281 12.49 MB/sec execute 87 sec latency 32.701 ms 2024-02-10T06:48:07.630 INFO:teuthology.orchestra.run.smithi008.stdout: 5 120670 12.47 MB/sec execute 88 sec latency 79.597 ms 2024-02-10T06:48:08.132 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:07 smithi008 ceph-mon[24646]: pgmap v33: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.1 MiB/s wr, 493 op/s 2024-02-10T06:48:08.132 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:48:08.132 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:07 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:08.132 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:07 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:07 smithi029 ceph-mon[27077]: pgmap v33: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.1 MiB/s wr, 493 op/s 2024-02-10T06:48:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:48:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:07 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:07 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:08.630 INFO:teuthology.orchestra.run.smithi008.stdout: 5 121016 12.46 MB/sec execute 89 sec latency 38.231 ms 2024-02-10T06:48:09.166 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:08 smithi008 ceph-mon[24646]: Reconfiguring node-exporter.smithi008 (dependencies changed)... 2024-02-10T06:48:09.166 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:08 smithi008 ceph-mon[24646]: Reconfiguring daemon node-exporter.smithi008 on smithi008 2024-02-10T06:48:09.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:08 smithi029 ceph-mon[27077]: Reconfiguring node-exporter.smithi008 (dependencies changed)... 2024-02-10T06:48:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:08 smithi029 ceph-mon[27077]: Reconfiguring daemon node-exporter.smithi008 on smithi008 2024-02-10T06:48:09.630 INFO:teuthology.orchestra.run.smithi008.stdout: 5 121816 12.50 MB/sec execute 90 sec latency 25.602 ms 2024-02-10T06:48:10.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:09 smithi008 ceph-mon[24646]: pgmap v34: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 856 op/s 2024-02-10T06:48:10.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:09 smithi029 ceph-mon[27077]: pgmap v34: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 856 op/s 2024-02-10T06:48:10.630 INFO:teuthology.orchestra.run.smithi008.stdout: 5 122538 12.56 MB/sec execute 91 sec latency 31.543 ms 2024-02-10T06:48:11.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 123084 12.56 MB/sec execute 92 sec latency 35.782 ms 2024-02-10T06:48:12.125 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:11 smithi008 ceph-mon[24646]: pgmap v35: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.8 MiB/s wr, 630 op/s 2024-02-10T06:48:12.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:11 smithi029 ceph-mon[27077]: pgmap v35: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.8 MiB/s wr, 630 op/s 2024-02-10T06:48:12.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 123783 12.68 MB/sec execute 93 sec latency 24.801 ms 2024-02-10T06:48:13.144 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:12 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:13.144 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:12 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:13.144 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:12 smithi008 ceph-mon[24646]: Reconfiguring prometheus.smithi008 (dependencies changed)... 2024-02-10T06:48:13.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:12 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:13.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:12 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:13.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:12 smithi029 ceph-mon[27077]: Reconfiguring prometheus.smithi008 (dependencies changed)... 2024-02-10T06:48:13.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 124188 12.69 MB/sec execute 94 sec latency 142.077 ms 2024-02-10T06:48:14.149 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:13 smithi008 ceph-mon[24646]: Reconfiguring daemon prometheus.smithi008 on smithi008 2024-02-10T06:48:14.149 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:13 smithi008 ceph-mon[24646]: pgmap v36: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.8 MiB/s wr, 630 op/s 2024-02-10T06:48:14.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:13 smithi029 ceph-mon[27077]: Reconfiguring daemon prometheus.smithi008 on smithi008 2024-02-10T06:48:14.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:13 smithi029 ceph-mon[27077]: pgmap v36: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.8 MiB/s wr, 630 op/s 2024-02-10T06:48:14.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 124542 12.68 MB/sec execute 95 sec latency 29.576 ms 2024-02-10T06:48:15.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 125240 12.70 MB/sec execute 96 sec latency 39.844 ms 2024-02-10T06:48:16.169 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:15 smithi008 ceph-mon[24646]: pgmap v37: 97 pgs: 97 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 992 op/s 2024-02-10T06:48:16.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:15 smithi029 ceph-mon[27077]: pgmap v37: 97 pgs: 97 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 992 op/s 2024-02-10T06:48:16.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 125982 12.76 MB/sec execute 97 sec latency 27.895 ms 2024-02-10T06:48:17.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 126429 12.70 MB/sec execute 98 sec latency 40.384 ms 2024-02-10T06:48:18.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:17 smithi008 ceph-mon[24646]: pgmap v38: 97 pgs: 97 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.4 MiB/s wr, 724 op/s 2024-02-10T06:48:18.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:17 smithi029 ceph-mon[27077]: pgmap v38: 97 pgs: 97 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.4 MiB/s wr, 724 op/s 2024-02-10T06:48:18.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 126864 12.76 MB/sec execute 99 sec latency 41.818 ms 2024-02-10T06:48:19.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 127351 12.76 MB/sec execute 100 sec latency 36.305 ms 2024-02-10T06:48:20.202 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:19 smithi008 ceph-mon[24646]: pgmap v39: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 15 KiB/s rd, 11 MiB/s wr, 1.04k op/s 2024-02-10T06:48:20.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:19 smithi029 ceph-mon[27077]: pgmap v39: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 15 KiB/s rd, 11 MiB/s wr, 1.04k op/s 2024-02-10T06:48:20.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 128108 12.87 MB/sec execute 101 sec latency 24.254 ms 2024-02-10T06:48:21.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 128598 12.85 MB/sec execute 102 sec latency 38.108 ms 2024-02-10T06:48:22.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:21 smithi008 ceph-mon[24646]: pgmap v40: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-02-10T06:48:22.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:21 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:48:22.435 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:21 smithi029 ceph-mon[27077]: pgmap v40: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-02-10T06:48:22.435 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:21 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:48:22.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 128883 12.78 MB/sec execute 103 sec latency 45.088 ms 2024-02-10T06:48:23.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 129203 12.74 MB/sec execute 104 sec latency 46.376 ms 2024-02-10T06:48:23.983 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:23 smithi008 ceph-mon[24646]: pgmap v41: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-02-10T06:48:24.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:23 smithi029 ceph-mon[27077]: pgmap v41: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-02-10T06:48:24.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 129616 12.71 MB/sec execute 105 sec latency 45.327 ms 2024-02-10T06:48:25.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 129967 12.64 MB/sec execute 106 sec latency 39.695 ms 2024-02-10T06:48:26.048 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:25 smithi008 ceph-mon[24646]: pgmap v42: 97 pgs: 97 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 15 KiB/s rd, 10 MiB/s wr, 975 op/s 2024-02-10T06:48:26.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:25 smithi029 ceph-mon[27077]: pgmap v42: 97 pgs: 97 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 15 KiB/s rd, 10 MiB/s wr, 975 op/s 2024-02-10T06:48:26.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 130226 12.60 MB/sec execute 107 sec latency 48.536 ms 2024-02-10T06:48:27.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 130540 12.61 MB/sec execute 108 sec latency 47.307 ms 2024-02-10T06:48:28.073 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:27 smithi008 ceph-mon[24646]: pgmap v43: 97 pgs: 97 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.3 MiB/s wr, 614 op/s 2024-02-10T06:48:28.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:27 smithi029 ceph-mon[27077]: pgmap v43: 97 pgs: 97 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.3 MiB/s wr, 614 op/s 2024-02-10T06:48:28.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 131062 12.64 MB/sec execute 109 sec latency 37.217 ms 2024-02-10T06:48:29.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 131355 12.61 MB/sec execute 110 sec latency 47.481 ms 2024-02-10T06:48:30.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:29 smithi008 ceph-mon[24646]: pgmap v44: 97 pgs: 97 active+clean; 341 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 818 op/s 2024-02-10T06:48:30.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:29 smithi029 ceph-mon[27077]: pgmap v44: 97 pgs: 97 active+clean; 341 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 818 op/s 2024-02-10T06:48:30.502 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:48:30.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 131616 12.57 MB/sec execute 111 sec latency 42.324 ms 2024-02-10T06:48:31.572 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:48:31.572 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 starting - - - - 2024-02-10T06:48:31.573 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (10m) 69s ago 10m 6405k - 17.2.0 e1d6a67b021e 5fa2e5bdd7cc 2024-02-10T06:48:31.573 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (9m) 73s ago 9m 6405k - 17.2.0 e1d6a67b021e 7527ea77b5cc 2024-02-10T06:48:31.573 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 starting - - - - 2024-02-10T06:48:31.573 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (4m) 69s ago 4m 36.7M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:48:31.573 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (4m) 73s ago 4m 13.0M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:48:31.573 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:9283 running (11m) 69s ago 11m 355M - 17.2.0 e1d6a67b021e b0d0b31e96f4 2024-02-10T06:48:31.573 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283 running (106s) 73s ago 9m 490M - 19.0.0-1361-g290212a1 c328ab9dcfc7 754f1049995b 2024-02-10T06:48:31.573 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (11m) 69s ago 12m 52.0M 2048M 17.2.0 e1d6a67b021e 061312907e4b 2024-02-10T06:48:31.573 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (9m) 73s ago 9m 41.7M 2048M 17.2.0 e1d6a67b021e a790a61584bf 2024-02-10T06:48:31.573 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (4m) 69s ago 4m 249M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:48:31.573 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (3m) 73s ago 3m 73.8M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:48:31.573 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 starting - - - - 2024-02-10T06:48:31.573 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (9m) 73s ago 9m 19.5M - 1.3.1 1dbe0e931976 6e5039760c8a 2024-02-10T06:48:31.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (9m) 69s ago 9m 365M 1937M 17.2.0 e1d6a67b021e 139b7a92bfc4 2024-02-10T06:48:31.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (8m) 69s ago 8m 272M 1937M 17.2.0 e1d6a67b021e 0f1119e17a52 2024-02-10T06:48:31.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (8m) 69s ago 8m 252M 1937M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:48:31.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (7m) 69s ago 7m 336M 1937M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:48:31.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (7m) 73s ago 7m 345M 2705M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:48:31.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (6m) 73s ago 6m 372M 2705M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:48:31.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (6m) 73s ago 6m 306M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:48:31.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (5m) 73s ago 5m 459M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:48:31.574 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (9m) 69s ago 9m 67.5M - 2.33.4 514e6a882f6e eedfd13fcab7 2024-02-10T06:48:31.632 INFO:teuthology.orchestra.run.smithi008.stdout: 5 132071 12.55 MB/sec execute 112 sec latency 42.395 ms 2024-02-10T06:48:32.128 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:31 smithi008 ceph-mon[24646]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:48:32.128 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:31 smithi008 ceph-mon[24646]: pgmap v45: 97 pgs: 97 active+clean; 341 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 MiB/s wr, 502 op/s 2024-02-10T06:48:32.128 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:31 smithi008 ceph-mon[24646]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:48:32.227 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:48:32.227 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:48:32.227 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:48:32.227 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:48:32.227 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:48:32.228 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T06:48:32.228 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T06:48:32.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:48:32.228 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:48:32.228 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:48:32.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:48:32.228 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:48:32.228 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:48:32.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:48:32.229 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:48:32.229 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-10T06:48:32.229 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T06:48:32.229 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:48:32.229 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:48:32.436 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:31 smithi029 ceph-mon[27077]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:48:32.436 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:31 smithi029 ceph-mon[27077]: pgmap v45: 97 pgs: 97 active+clean; 341 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 MiB/s wr, 502 op/s 2024-02-10T06:48:32.436 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:31 smithi029 ceph-mon[27077]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:48:32.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 132438 12.50 MB/sec execute 113 sec latency 47.609 ms 2024-02-10T06:48:32.777 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:48:32.777 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:48:32.777 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:48:32.777 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:48:32.777 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-02-10T06:48:32.777 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "1/23 daemons upgraded", 2024-02-10T06:48:32.777 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-02-10T06:48:32.777 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:48:32.777 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:48:33.143 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:33 smithi008 ceph-mon[24646]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:48:33.143 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:33 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/3916376870' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:48:33.353 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:48:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:33 smithi029 ceph-mon[27077]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:48:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:33 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/3916376870' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:48:33.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 132748 12.46 MB/sec execute 114 sec latency 50.965 ms 2024-02-10T06:48:34.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:34 smithi008 ceph-mon[24646]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:48:34.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:34 smithi008 ceph-mon[24646]: pgmap v46: 97 pgs: 97 active+clean; 341 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 MiB/s wr, 502 op/s 2024-02-10T06:48:34.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:34 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/2102755806' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:48:34.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:34 smithi029 ceph-mon[27077]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:48:34.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:34 smithi029 ceph-mon[27077]: pgmap v46: 97 pgs: 97 active+clean; 341 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 MiB/s wr, 502 op/s 2024-02-10T06:48:34.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:34 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/2102755806' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:48:34.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 133160 12.43 MB/sec execute 115 sec latency 54.801 ms 2024-02-10T06:48:35.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 133538 12.38 MB/sec execute 116 sec latency 36.760 ms 2024-02-10T06:48:36.197 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:36 smithi008 ceph-mon[24646]: pgmap v47: 97 pgs: 97 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 734 op/s 2024-02-10T06:48:36.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:36 smithi029 ceph-mon[27077]: pgmap v47: 97 pgs: 97 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 734 op/s 2024-02-10T06:48:36.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 133757 12.34 MB/sec execute 117 sec latency 65.788 ms 2024-02-10T06:48:37.216 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:37 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:48:37.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:37 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:48:37.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 134083 12.35 MB/sec execute 118 sec latency 66.497 ms 2024-02-10T06:48:38.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:38 smithi008 ceph-mon[24646]: pgmap v48: 97 pgs: 97 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.4 MiB/s wr, 435 op/s 2024-02-10T06:48:38.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:38 smithi029 ceph-mon[27077]: pgmap v48: 97 pgs: 97 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.4 MiB/s wr, 435 op/s 2024-02-10T06:48:38.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 134669 12.40 MB/sec execute 119 sec latency 36.643 ms 2024-02-10T06:48:39.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:39 smithi008 ceph-mon[24646]: mgrmap e27: smithi029.fuvrtf(active, since 92s), standbys: smithi008.pxifbt 2024-02-10T06:48:39.270 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:39 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:39.270 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:39 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:39.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:39 smithi029 ceph-mon[27077]: mgrmap e27: smithi029.fuvrtf(active, since 92s), standbys: smithi008.pxifbt 2024-02-10T06:48:39.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:39 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:39.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:39 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:39.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 134977 12.37 MB/sec execute 120 sec latency 47.406 ms 2024-02-10T06:48:40.037 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:40 smithi008 ceph-mon[24646]: Reconfiguring node-exporter.smithi029 (dependencies changed)... 2024-02-10T06:48:40.037 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:40 smithi008 ceph-mon[24646]: Reconfiguring daemon node-exporter.smithi029 on smithi029 2024-02-10T06:48:40.037 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:40 smithi008 ceph-mon[24646]: pgmap v49: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.4 MiB/s wr, 640 op/s 2024-02-10T06:48:40.314 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:40 smithi029 ceph-mon[27077]: Reconfiguring node-exporter.smithi029 (dependencies changed)... 2024-02-10T06:48:40.314 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:40 smithi029 ceph-mon[27077]: Reconfiguring daemon node-exporter.smithi029 on smithi029 2024-02-10T06:48:40.314 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:40 smithi029 ceph-mon[27077]: pgmap v49: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.4 MiB/s wr, 640 op/s 2024-02-10T06:48:40.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 135506 12.38 MB/sec execute 121 sec latency 29.655 ms 2024-02-10T06:48:41.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 135978 12.36 MB/sec execute 122 sec latency 33.299 ms 2024-02-10T06:48:41.813 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:41.814 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:41.814 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:48:41.814 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:48:41.814 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:48:41.814 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:48:41.814 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:41.814 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:48:41.814 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:48:41.814 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi008.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:48:41.814 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi008.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:48:41.814 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:41.814 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:48:41.814 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:48:41.814 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi008.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:48:41.815 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi008.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:48:41.815 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:41.815 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:48:41.815 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: Upgrade: Need to upgrade myself (mgr.smithi029.fuvrtf) 2024-02-10T06:48:41.815 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: Upgrade: Need to upgrade myself (mgr.smithi029.fuvrtf) 2024-02-10T06:48:41.815 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:41 smithi008 ceph-mon[24646]: pgmap v50: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.4 MiB/s wr, 435 op/s 2024-02-10T06:48:42.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:42.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:48:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:48:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:48:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:48:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:48:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:48:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi008.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:48:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi008.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:48:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:48:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:48:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi008.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:48:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi008.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:48:42.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:42.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:48:42.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: Upgrade: Need to upgrade myself (mgr.smithi029.fuvrtf) 2024-02-10T06:48:42.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: Upgrade: Need to upgrade myself (mgr.smithi029.fuvrtf) 2024-02-10T06:48:42.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:41 smithi029 ceph-mon[27077]: pgmap v50: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.4 MiB/s wr, 435 op/s 2024-02-10T06:48:42.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 136224 12.33 MB/sec execute 123 sec latency 40.427 ms 2024-02-10T06:48:43.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 136568 12.29 MB/sec execute 124 sec latency 72.949 ms 2024-02-10T06:48:44.163 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:43 smithi008 ceph-mon[24646]: pgmap v51: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.4 MiB/s wr, 435 op/s 2024-02-10T06:48:44.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:43 smithi029 ceph-mon[27077]: pgmap v51: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.4 MiB/s wr, 435 op/s 2024-02-10T06:48:44.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 136895 12.22 MB/sec execute 125 sec latency 86.074 ms 2024-02-10T06:48:45.172 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:45 smithi008 ceph-mon[24646]: Upgrade: Updating mgr.smithi008.pxifbt 2024-02-10T06:48:45.172 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:45 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:45.172 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:45 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.pxifbt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:48:45.172 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:45 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.pxifbt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:48:45.172 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:45 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:48:45.172 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:45 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:48:45.172 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:45 smithi008 ceph-mon[24646]: Deploying daemon mgr.smithi008.pxifbt on smithi008 2024-02-10T06:48:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:45 smithi029 ceph-mon[27077]: Upgrade: Updating mgr.smithi008.pxifbt 2024-02-10T06:48:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:45 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:45 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.pxifbt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:48:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:45 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.pxifbt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:48:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:45 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:48:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:45 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:48:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:45 smithi029 ceph-mon[27077]: Deploying daemon mgr.smithi008.pxifbt on smithi008 2024-02-10T06:48:45.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 137154 12.19 MB/sec execute 126 sec latency 63.913 ms 2024-02-10T06:48:46.191 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:46 smithi008 ceph-mon[24646]: pgmap v52: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 696 op/s 2024-02-10T06:48:46.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:46 smithi029 ceph-mon[27077]: pgmap v52: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 696 op/s 2024-02-10T06:48:46.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 137488 12.21 MB/sec execute 127 sec latency 60.633 ms 2024-02-10T06:48:47.634 INFO:teuthology.orchestra.run.smithi008.stdout: 5 137995 12.21 MB/sec execute 128 sec latency 53.288 ms 2024-02-10T06:48:48.211 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:48 smithi008 ceph-mon[24646]: pgmap v53: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.6 MiB/s wr, 464 op/s 2024-02-10T06:48:48.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:48 smithi029 ceph-mon[27077]: pgmap v53: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.6 MiB/s wr, 464 op/s 2024-02-10T06:48:48.633 INFO:teuthology.orchestra.run.smithi008.stdout: 5 138444 12.24 MB/sec execute 129 sec latency 46.366 ms 2024-02-10T06:48:49.634 INFO:teuthology.orchestra.run.smithi008.stdout: 5 138787 12.21 MB/sec execute 130 sec latency 42.076 ms 2024-02-10T06:48:50.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:50 smithi008 ceph-mon[24646]: pgmap v54: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.8 MiB/s wr, 667 op/s 2024-02-10T06:48:50.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:50 smithi029 ceph-mon[27077]: pgmap v54: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.8 MiB/s wr, 667 op/s 2024-02-10T06:48:50.634 INFO:teuthology.orchestra.run.smithi008.stdout: 5 139228 12.19 MB/sec execute 131 sec latency 49.103 ms 2024-02-10T06:48:51.634 INFO:teuthology.orchestra.run.smithi008.stdout: 5 139523 12.15 MB/sec execute 132 sec latency 48.781 ms 2024-02-10T06:48:52.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:52 smithi008 ceph-mon[24646]: pgmap v55: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.8 MiB/s wr, 463 op/s 2024-02-10T06:48:52.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:52 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:48:52.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:52 smithi029 ceph-mon[27077]: pgmap v55: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.8 MiB/s wr, 463 op/s 2024-02-10T06:48:52.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:52 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:48:52.634 INFO:teuthology.orchestra.run.smithi008.stdout: 5 139866 12.13 MB/sec execute 133 sec latency 36.462 ms 2024-02-10T06:48:53.634 INFO:teuthology.orchestra.run.smithi008.stdout: 5 140212 12.10 MB/sec execute 134 sec latency 53.135 ms 2024-02-10T06:48:54.046 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:53 smithi008 ceph-mon[24646]: pgmap v56: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.8 MiB/s wr, 463 op/s 2024-02-10T06:48:54.046 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:53 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:54.046 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:53 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:54.046 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:53 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:48:54.046 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:53 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:54.047 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:53 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:54.047 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:53 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:54.047 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:53 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:54.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:53 smithi029 ceph-mon[27077]: pgmap v56: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.8 MiB/s wr, 463 op/s 2024-02-10T06:48:54.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:53 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:54.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:53 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:54.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:53 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:48:54.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:53 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:54.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:53 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:54.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:53 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:54.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:53 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:48:54.634 INFO:teuthology.orchestra.run.smithi008.stdout: 5 140565 12.04 MB/sec execute 135 sec latency 40.650 ms 2024-02-10T06:48:55.634 INFO:teuthology.orchestra.run.smithi008.stdout: 5 141071 12.11 MB/sec execute 136 sec latency 33.079 ms 2024-02-10T06:48:56.080 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:55 smithi008 ceph-mon[24646]: pgmap v57: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 703 op/s 2024-02-10T06:48:56.287 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:55 smithi029 ceph-mon[27077]: pgmap v57: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 703 op/s 2024-02-10T06:48:56.634 INFO:teuthology.orchestra.run.smithi008.stdout: 5 141672 12.14 MB/sec execute 137 sec latency 40.183 ms 2024-02-10T06:48:57.634 INFO:teuthology.orchestra.run.smithi008.stdout: 5 142171 12.16 MB/sec execute 138 sec latency 25.104 ms 2024-02-10T06:48:58.093 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:48:57 smithi008 ceph-mon[24646]: pgmap v58: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 443 op/s 2024-02-10T06:48:58.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:48:57 smithi029 ceph-mon[27077]: pgmap v58: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 443 op/s 2024-02-10T06:48:58.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 142824 12.18 MB/sec execute 139 sec latency 41.862 ms 2024-02-10T06:48:59.634 INFO:teuthology.orchestra.run.smithi008.stdout: 5 143257 12.19 MB/sec execute 140 sec latency 37.776 ms 2024-02-10T06:49:00.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:00 smithi008 ceph-mon[24646]: pgmap v59: 97 pgs: 97 active+clean; 348 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 7.5 MiB/s wr, 716 op/s 2024-02-10T06:49:00.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:00 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:00.107 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:00 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:00.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:00 smithi029 ceph-mon[27077]: pgmap v59: 97 pgs: 97 active+clean; 348 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 7.5 MiB/s wr, 716 op/s 2024-02-10T06:49:00.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:00 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:00.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:00 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:00.634 INFO:teuthology.orchestra.run.smithi008.stdout: 5 143767 12.17 MB/sec execute 141 sec latency 31.119 ms 2024-02-10T06:49:01.634 INFO:teuthology.orchestra.run.smithi008.stdout: 5 144282 12.17 MB/sec execute 142 sec latency 78.796 ms 2024-02-10T06:49:02.140 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:02 smithi008 ceph-mon[24646]: pgmap v60: 97 pgs: 97 active+clean; 348 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.4 MiB/s wr, 514 op/s 2024-02-10T06:49:02.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:02 smithi029 ceph-mon[27077]: pgmap v60: 97 pgs: 97 active+clean; 348 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.4 MiB/s wr, 514 op/s 2024-02-10T06:49:02.634 INFO:teuthology.orchestra.run.smithi008.stdout: 5 144754 12.21 MB/sec execute 143 sec latency 80.282 ms 2024-02-10T06:49:03.634 INFO:teuthology.orchestra.run.smithi008.stdout: 5 145263 12.21 MB/sec execute 144 sec latency 30.570 ms 2024-02-10T06:49:03.929 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:49:04.204 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:04 smithi008 ceph-mon[24646]: pgmap v61: 97 pgs: 97 active+clean; 348 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.4 MiB/s wr, 514 op/s 2024-02-10T06:49:04.204 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:04 smithi008 ceph-mon[24646]: Standby manager daemon smithi008.pxifbt restarted 2024-02-10T06:49:04.204 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:04 smithi008 ceph-mon[24646]: Standby manager daemon smithi008.pxifbt started 2024-02-10T06:49:04.204 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:04 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.pxifbt/crt"}]: dispatch 2024-02-10T06:49:04.204 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:04 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:49:04.205 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:04 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.pxifbt/key"}]: dispatch 2024-02-10T06:49:04.205 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:04 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:49:04.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:04 smithi029 ceph-mon[27077]: pgmap v61: 97 pgs: 97 active+clean; 348 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.4 MiB/s wr, 514 op/s 2024-02-10T06:49:04.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:04 smithi029 ceph-mon[27077]: Standby manager daemon smithi008.pxifbt restarted 2024-02-10T06:49:04.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:04 smithi029 ceph-mon[27077]: Standby manager daemon smithi008.pxifbt started 2024-02-10T06:49:04.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:04 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.pxifbt/crt"}]: dispatch 2024-02-10T06:49:04.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:04 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:49:04.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:04 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.pxifbt/key"}]: dispatch 2024-02-10T06:49:04.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:04 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:49:04.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 145641 12.22 MB/sec execute 145 sec latency 51.835 ms 2024-02-10T06:49:04.978 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:49:04.978 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (83s) 0s ago 11m 16.7M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:49:04.978 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (11m) 0s ago 11m 6405k - 17.2.0 e1d6a67b021e 5fa2e5bdd7cc 2024-02-10T06:49:04.978 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (10m) 5s ago 10m 6405k - 17.2.0 e1d6a67b021e 7527ea77b5cc 2024-02-10T06:49:04.978 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (58s) 0s ago 10m 42.4M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:49:04.978 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (4m) 0s ago 4m 37.4M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:49:04.978 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (4m) 5s ago 4m 13.4M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:49:04.978 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (13s) 0s ago 12m 94.4M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:49:04.978 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283 running (2m) 5s ago 10m 503M - 19.0.0-1361-g290212a1 c328ab9dcfc7 754f1049995b 2024-02-10T06:49:04.978 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (12m) 0s ago 12m 53.2M 2048M 17.2.0 e1d6a67b021e 061312907e4b 2024-02-10T06:49:04.979 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (10m) 5s ago 10m 45.1M 2048M 17.2.0 e1d6a67b021e a790a61584bf 2024-02-10T06:49:04.979 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (4m) 0s ago 4m 268M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:49:04.979 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (4m) 5s ago 4m 79.8M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:49:04.979 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (54s) 0s ago 10m 9315k - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:49:04.979 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (24s) 5s ago 10m 10.2M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:49:04.979 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (9m) 0s ago 9m 481M 1937M 17.2.0 e1d6a67b021e 139b7a92bfc4 2024-02-10T06:49:04.979 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (9m) 0s ago 9m 418M 1937M 17.2.0 e1d6a67b021e 0f1119e17a52 2024-02-10T06:49:04.979 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (8m) 0s ago 8m 306M 1937M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:49:04.979 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (8m) 0s ago 8m 450M 1937M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:49:04.979 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (7m) 5s ago 7m 481M 2705M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:49:04.979 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (7m) 5s ago 7m 537M 2705M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:49:04.979 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (6m) 5s ago 6m 441M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:49:04.979 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (6m) 5s ago 6m 352M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:49:04.979 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (27s) 0s ago 9m 53.2M - 2.33.4 514e6a882f6e 2901a03ac60a 2024-02-10T06:49:05.189 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:05 smithi008 ceph-mon[24646]: from='client.24553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:49:05.189 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:05 smithi008 ceph-mon[24646]: mgrmap e28: smithi029.fuvrtf(active, since 118s), standbys: smithi008.pxifbt 2024-02-10T06:49:05.189 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:05 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:05.189 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:05 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:05.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:05 smithi029 ceph-mon[27077]: from='client.24553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:49:05.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:05 smithi029 ceph-mon[27077]: mgrmap e28: smithi029.fuvrtf(active, since 118s), standbys: smithi008.pxifbt 2024-02-10T06:49:05.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:05 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:05.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:05 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:05.634 INFO:teuthology.orchestra.run.smithi008.stdout: 5 146081 12.21 MB/sec execute 146 sec latency 61.051 ms 2024-02-10T06:49:05.752 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:49:05.752 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:49:05.752 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:49:05.752 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:49:05.752 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:49:05.752 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:49:05.753 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:49:05.753 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:49:05.753 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:49:05.753 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:49:05.753 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:49:05.753 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:49:05.753 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:49:05.753 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:49:05.753 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-02-10T06:49:05.753 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:49:05.753 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:49:05.753 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:49:06.207 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:06 smithi008 ceph-mon[24646]: from='client.24557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:49:06.207 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:06 smithi008 ceph-mon[24646]: pgmap v62: 97 pgs: 97 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 822 op/s 2024-02-10T06:49:06.207 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:06 smithi008 ceph-mon[24646]: from='client.24561 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:49:06.207 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:06 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/2381365487' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:49:06.207 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:06 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:06.207 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:06 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:06.293 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:49:06.293 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:49:06.294 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:49:06.294 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:49:06.294 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:49:06.294 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-02-10T06:49:06.294 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:49:06.294 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "2/23 daemons upgraded", 2024-02-10T06:49:06.294 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading mgr daemons", 2024-02-10T06:49:06.294 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:49:06.294 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:49:06.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:06 smithi029 ceph-mon[27077]: from='client.24557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:49:06.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:06 smithi029 ceph-mon[27077]: pgmap v62: 97 pgs: 97 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 822 op/s 2024-02-10T06:49:06.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:06 smithi029 ceph-mon[27077]: from='client.24561 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:49:06.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:06 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/2381365487' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:49:06.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:06 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:06.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:06 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:06.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 146469 12.18 MB/sec execute 147 sec latency 62.110 ms 2024-02-10T06:49:06.995 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:49:07.216 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:07 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:49:07.216 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:07 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/3531215895' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:49:07.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:07 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:49:07.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:07 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/3531215895' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:49:07.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 146789 12.16 MB/sec execute 148 sec latency 37.541 ms 2024-02-10T06:49:08.256 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:08 smithi008 ceph-mon[24646]: from='client.14830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:49:08.256 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:08 smithi008 ceph-mon[24646]: pgmap v63: 97 pgs: 97 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.1 MiB/s wr, 581 op/s 2024-02-10T06:49:08.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:08 smithi029 ceph-mon[27077]: from='client.14830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:49:08.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:08 smithi029 ceph-mon[27077]: pgmap v63: 97 pgs: 97 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.1 MiB/s wr, 581 op/s 2024-02-10T06:49:08.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 147315 12.16 MB/sec execute 149 sec latency 111.674 ms 2024-02-10T06:49:09.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 147832 12.16 MB/sec execute 150 sec latency 31.929 ms 2024-02-10T06:49:10.216 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:10 smithi008 ceph-mon[24646]: pgmap v64: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 840 op/s 2024-02-10T06:49:10.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:10 smithi029 ceph-mon[27077]: pgmap v64: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 840 op/s 2024-02-10T06:49:10.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 148284 12.20 MB/sec execute 151 sec latency 58.882 ms 2024-02-10T06:49:11.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 148790 12.20 MB/sec execute 152 sec latency 61.153 ms 2024-02-10T06:49:12.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:12 smithi008 ceph-mon[24646]: pgmap v65: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.5 MiB/s wr, 567 op/s 2024-02-10T06:49:12.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:12 smithi029 ceph-mon[27077]: pgmap v65: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.5 MiB/s wr, 567 op/s 2024-02-10T06:49:12.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 149092 12.19 MB/sec execute 153 sec latency 81.404 ms 2024-02-10T06:49:13.445 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:13 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:13.445 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:13 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:13.445 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:13 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:49:13.445 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:13 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:49:13.445 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:13 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:13.446 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:13 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:13.446 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:13 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:49:13.446 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:13 smithi008 ceph-mon[24646]: Upgrade: Need to upgrade myself (mgr.smithi029.fuvrtf) 2024-02-10T06:49:13.446 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:13 smithi008 ceph-mon[24646]: Upgrade: Need to upgrade myself (mgr.smithi029.fuvrtf) 2024-02-10T06:49:13.446 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:13 smithi008 ceph-mon[24646]: Failing over to other MGR 2024-02-10T06:49:13.446 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:13 smithi008 ceph-mon[24646]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mgr fail", "who": "smithi029.fuvrtf"}]: dispatch 2024-02-10T06:49:13.446 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:13 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mgr fail", "who": "smithi029.fuvrtf"}]: dispatch 2024-02-10T06:49:13.446 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:13 smithi008 ceph-mon[24646]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T06:49:13.446 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:13 smithi008 ceph-mon[24646]: pgmap v67: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.7 MiB/s wr, 680 op/s 2024-02-10T06:49:13.516 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:13 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:13.516 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:13 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:13.516 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:13 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:49:13.516 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:13 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:49:13.516 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:13 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:13.517 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:13 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' 2024-02-10T06:49:13.517 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:13 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:49:13.517 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:13 smithi029 ceph-mon[27077]: Upgrade: Need to upgrade myself (mgr.smithi029.fuvrtf) 2024-02-10T06:49:13.517 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:13 smithi029 ceph-mon[27077]: Upgrade: Need to upgrade myself (mgr.smithi029.fuvrtf) 2024-02-10T06:49:13.517 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:13 smithi029 ceph-mon[27077]: Failing over to other MGR 2024-02-10T06:49:13.517 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:13 smithi029 ceph-mon[27077]: from='mgr.24459 172.21.15.29:0/2891630870' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mgr fail", "who": "smithi029.fuvrtf"}]: dispatch 2024-02-10T06:49:13.517 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:13 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "mgr fail", "who": "smithi029.fuvrtf"}]: dispatch 2024-02-10T06:49:13.517 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:13 smithi029 ceph-mon[27077]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T06:49:13.517 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:13 smithi029 ceph-mon[27077]: pgmap v67: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.7 MiB/s wr, 680 op/s 2024-02-10T06:49:13.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 149336 12.16 MB/sec execute 154 sec latency 35.937 ms 2024-02-10T06:49:14.479 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd='[{"prefix": "mgr fail", "who": "smithi029.fuvrtf"}]': finished 2024-02-10T06:49:14.479 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: mgrmap e29: smithi008.pxifbt(active, starting, since 0.861698s) 2024-02-10T06:49:14.479 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:49:14.479 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:49:14.479 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.vfwcte"}]: dispatch 2024-02-10T06:49:14.479 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi029.bwmcyt"}]: dispatch 2024-02-10T06:49:14.479 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr metadata", "who": "smithi008.pxifbt", "id": "smithi008.pxifbt"}]: dispatch 2024-02-10T06:49:14.479 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:49:14.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:49:14.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:49:14.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:49:14.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:49:14.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:49:14.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:49:14.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:49:14.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:49:14.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:49:14.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:49:14.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: Manager daemon smithi008.pxifbt is now available 2024-02-10T06:49:14.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:49:14.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:49:14.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:49:14.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:49:14.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:14 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:49:14.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 149588 12.12 MB/sec execute 155 sec latency 72.873 ms 2024-02-10T06:49:14.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.24459 ' entity='mgr.smithi029.fuvrtf' cmd='[{"prefix": "mgr fail", "who": "smithi029.fuvrtf"}]': finished 2024-02-10T06:49:14.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: mgrmap e29: smithi008.pxifbt(active, starting, since 0.861698s) 2024-02-10T06:49:14.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:49:14.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:49:14.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.vfwcte"}]: dispatch 2024-02-10T06:49:14.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi029.bwmcyt"}]: dispatch 2024-02-10T06:49:14.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr metadata", "who": "smithi008.pxifbt", "id": "smithi008.pxifbt"}]: dispatch 2024-02-10T06:49:14.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:49:14.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:49:14.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:49:14.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:49:14.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:49:14.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:49:14.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:49:14.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:49:14.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:49:14.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:49:14.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:49:14.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: Manager daemon smithi008.pxifbt is now available 2024-02-10T06:49:14.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:49:14.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:49:14.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:49:14.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:49:14.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:14 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:49:15.373 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:15 smithi029 ceph-mon[27077]: mgrmap e30: smithi008.pxifbt(active, since 1.88598s) 2024-02-10T06:49:15.373 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:15 smithi029 ceph-mon[27077]: pgmap v3: 97 pgs: 97 active+clean; 346 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-02-10T06:49:15.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:15 smithi008 ceph-mon[24646]: mgrmap e30: smithi008.pxifbt(active, since 1.88598s) 2024-02-10T06:49:15.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:15 smithi008 ceph-mon[24646]: pgmap v3: 97 pgs: 97 active+clean; 346 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-02-10T06:49:15.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 150074 12.10 MB/sec execute 156 sec latency 39.088 ms 2024-02-10T06:49:16.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:16 smithi008 ceph-mon[24646]: [10/Feb/2024:06:49:15] ENGINE Bus STARTING 2024-02-10T06:49:16.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:16 smithi008 ceph-mon[24646]: [10/Feb/2024:06:49:15] ENGINE Serving on https://172.21.15.8:7150 2024-02-10T06:49:16.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:16 smithi008 ceph-mon[24646]: [10/Feb/2024:06:49:15] ENGINE Serving on http://172.21.15.8:8765 2024-02-10T06:49:16.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:16 smithi008 ceph-mon[24646]: [10/Feb/2024:06:49:15] ENGINE Bus STARTED 2024-02-10T06:49:16.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 150457 12.10 MB/sec execute 157 sec latency 43.967 ms 2024-02-10T06:49:16.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:16 smithi029 ceph-mon[27077]: [10/Feb/2024:06:49:15] ENGINE Bus STARTING 2024-02-10T06:49:16.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:16 smithi029 ceph-mon[27077]: [10/Feb/2024:06:49:15] ENGINE Serving on https://172.21.15.8:7150 2024-02-10T06:49:16.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:16 smithi029 ceph-mon[27077]: [10/Feb/2024:06:49:15] ENGINE Serving on http://172.21.15.8:8765 2024-02-10T06:49:16.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:16 smithi029 ceph-mon[27077]: [10/Feb/2024:06:49:15] ENGINE Bus STARTED 2024-02-10T06:49:17.498 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:17 smithi008 ceph-mon[24646]: mgrmap e31: smithi008.pxifbt(active, since 3s) 2024-02-10T06:49:17.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 151001 12.09 MB/sec execute 158 sec latency 83.954 ms 2024-02-10T06:49:17.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:17 smithi029 ceph-mon[27077]: mgrmap e31: smithi008.pxifbt(active, since 3s) 2024-02-10T06:49:18.531 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:18 smithi008 ceph-mon[24646]: pgmap v5: 97 pgs: 97 active+clean; 346 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-02-10T06:49:18.531 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:18 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:18.532 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:18 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:18.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 151426 12.08 MB/sec execute 159 sec latency 33.782 ms 2024-02-10T06:49:18.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:18 smithi029 ceph-mon[27077]: pgmap v5: 97 pgs: 97 active+clean; 346 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-02-10T06:49:18.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:18 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:18.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:18 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:19.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 151860 12.12 MB/sec execute 160 sec latency 35.772 ms 2024-02-10T06:49:20.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 152460 12.14 MB/sec execute 161 sec latency 40.722 ms 2024-02-10T06:49:20.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:20 smithi008 ceph-mon[24646]: pgmap v6: 97 pgs: 97 active+clean; 346 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-02-10T06:49:20.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:20 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:20.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:20 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:20.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:20 smithi029 ceph-mon[27077]: pgmap v6: 97 pgs: 97 active+clean; 346 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-02-10T06:49:20.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:20 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:20.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:20 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:21.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 152833 12.15 MB/sec execute 162 sec latency 51.992 ms 2024-02-10T06:49:22.569 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:22 smithi008 ceph-mon[24646]: pgmap v7: 97 pgs: 97 active+clean; 342 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 36 KiB/s rd, 4.5 MiB/s wr, 489 op/s 2024-02-10T06:49:22.569 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:22 smithi008 ceph-mon[24646]: Standby manager daemon smithi029.fuvrtf started 2024-02-10T06:49:22.569 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:22 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.29:0/2071561480' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/crt"}]: dispatch 2024-02-10T06:49:22.569 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:22 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.29:0/2071561480' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:49:22.569 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:22 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.29:0/2071561480' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/key"}]: dispatch 2024-02-10T06:49:22.569 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:22 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.29:0/2071561480' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:49:22.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 153262 12.11 MB/sec execute 163 sec latency 52.210 ms 2024-02-10T06:49:22.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:22 smithi029 ceph-mon[27077]: pgmap v7: 97 pgs: 97 active+clean; 342 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 36 KiB/s rd, 4.5 MiB/s wr, 489 op/s 2024-02-10T06:49:22.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:22 smithi029 ceph-mon[27077]: Standby manager daemon smithi029.fuvrtf started 2024-02-10T06:49:22.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:22 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/2071561480' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/crt"}]: dispatch 2024-02-10T06:49:22.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:22 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/2071561480' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:49:22.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:22 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/2071561480' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/key"}]: dispatch 2024-02-10T06:49:22.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:22 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/2071561480' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:49:23.580 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:23 smithi008 ceph-mon[24646]: mgrmap e32: smithi008.pxifbt(active, since 10s), standbys: smithi029.fuvrtf 2024-02-10T06:49:23.580 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:23 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr metadata", "who": "smithi029.fuvrtf", "id": "smithi029.fuvrtf"}]: dispatch 2024-02-10T06:49:23.580 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:23 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:23.580 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:23 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:23.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 153602 12.09 MB/sec execute 164 sec latency 70.669 ms 2024-02-10T06:49:23.769 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:23 smithi029 ceph-mon[27077]: mgrmap e32: smithi008.pxifbt(active, since 10s), standbys: smithi029.fuvrtf 2024-02-10T06:49:23.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:23 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr metadata", "who": "smithi029.fuvrtf", "id": "smithi029.fuvrtf"}]: dispatch 2024-02-10T06:49:23.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:23 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:23.770 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:23 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:24.591 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:24 smithi008 ceph-mon[24646]: pgmap v8: 97 pgs: 97 active+clean; 342 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 3.5 MiB/s wr, 379 op/s 2024-02-10T06:49:24.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 153865 12.07 MB/sec execute 165 sec latency 59.326 ms 2024-02-10T06:49:24.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:24 smithi029 ceph-mon[27077]: pgmap v8: 97 pgs: 97 active+clean; 342 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 3.5 MiB/s wr, 379 op/s 2024-02-10T06:49:25.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 154487 12.08 MB/sec execute 166 sec latency 64.095 ms 2024-02-10T06:49:25.762 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:25 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:25.762 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:25 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:25 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:25 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:26.635 INFO:teuthology.orchestra.run.smithi008.stdout: 5 155003 12.08 MB/sec execute 167 sec latency 32.542 ms 2024-02-10T06:49:26.858 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:26 smithi008 ceph-mon[24646]: pgmap v9: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.9 MiB/s wr, 592 op/s 2024-02-10T06:49:27.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:26 smithi029 ceph-mon[27077]: pgmap v9: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.9 MiB/s wr, 592 op/s 2024-02-10T06:49:27.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 155539 12.12 MB/sec execute 168 sec latency 42.329 ms 2024-02-10T06:49:28.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 156101 12.16 MB/sec execute 169 sec latency 41.369 ms 2024-02-10T06:49:28.876 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:28 smithi008 ceph-mon[24646]: pgmap v10: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 5.4 MiB/s wr, 539 op/s 2024-02-10T06:49:28.876 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:28 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:49:29.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:28 smithi029 ceph-mon[27077]: pgmap v10: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 5.4 MiB/s wr, 539 op/s 2024-02-10T06:49:29.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:28 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:49:29.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 156444 12.14 MB/sec execute 170 sec latency 51.821 ms 2024-02-10T06:49:30.142 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:29 smithi008 ceph-mon[24646]: pgmap v11: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 5.4 MiB/s wr, 539 op/s 2024-02-10T06:49:30.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:29 smithi029 ceph-mon[27077]: pgmap v11: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 5.4 MiB/s wr, 539 op/s 2024-02-10T06:49:30.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 156890 12.11 MB/sec execute 171 sec latency 74.948 ms 2024-02-10T06:49:31.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 157182 12.08 MB/sec execute 172 sec latency 74.965 ms 2024-02-10T06:49:32.432 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:32 smithi008 ceph-mon[24646]: pgmap v12: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 8.4 MiB/s wr, 839 op/s 2024-02-10T06:49:32.432 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:32 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:32.432 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:32 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:32.432 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:32 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:49:32.432 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:32 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:49:32.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 157548 12.09 MB/sec execute 173 sec latency 40.429 ms 2024-02-10T06:49:32.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:32 smithi029 ceph-mon[27077]: pgmap v12: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 8.4 MiB/s wr, 839 op/s 2024-02-10T06:49:32.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:32 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:32.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:32 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:32.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:32 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:49:32.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:32 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:49:33.445 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:33 smithi008 ceph-mon[24646]: Updating smithi008:/etc/ceph/ceph.conf 2024-02-10T06:49:33.445 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:33 smithi008 ceph-mon[24646]: Updating smithi029:/etc/ceph/ceph.conf 2024-02-10T06:49:33.445 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:33 smithi008 ceph-mon[24646]: Updating smithi029:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:49:33.445 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:33 smithi008 ceph-mon[24646]: Updating smithi008:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:49:33.445 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:33 smithi008 ceph-mon[24646]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:49:33.445 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:33 smithi008 ceph-mon[24646]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:49:33.446 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:33 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:33.446 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:33 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:33.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 158061 12.07 MB/sec execute 174 sec latency 41.565 ms 2024-02-10T06:49:33.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:33 smithi029 ceph-mon[27077]: Updating smithi008:/etc/ceph/ceph.conf 2024-02-10T06:49:33.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:33 smithi029 ceph-mon[27077]: Updating smithi029:/etc/ceph/ceph.conf 2024-02-10T06:49:33.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:33 smithi029 ceph-mon[27077]: Updating smithi029:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:49:33.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:33 smithi029 ceph-mon[27077]: Updating smithi008:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:49:33.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:33 smithi029 ceph-mon[27077]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:49:33.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:33 smithi029 ceph-mon[27077]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:49:33.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:33 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:33.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:33 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:34.474 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:34 smithi008 ceph-mon[24646]: Updating smithi029:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:49:34.474 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:34 smithi008 ceph-mon[24646]: Updating smithi008:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:49:34.474 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:34 smithi008 ceph-mon[24646]: pgmap v13: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.9 MiB/s wr, 557 op/s 2024-02-10T06:49:34.474 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:34 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:34.474 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:34 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:34.474 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:34 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:34.474 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:34 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:34.474 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:34 smithi008 ceph-mon[24646]: Reconfiguring prometheus.smithi008 (dependencies changed)... 2024-02-10T06:49:34.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 158500 12.06 MB/sec execute 175 sec latency 58.481 ms 2024-02-10T06:49:34.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:34 smithi029 ceph-mon[27077]: Updating smithi029:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:49:34.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:34 smithi029 ceph-mon[27077]: Updating smithi008:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:49:34.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:34 smithi029 ceph-mon[27077]: pgmap v13: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.9 MiB/s wr, 557 op/s 2024-02-10T06:49:34.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:34 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:34.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:34 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:34.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:34 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:34.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:34 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:34.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:34 smithi029 ceph-mon[27077]: Reconfiguring prometheus.smithi008 (dependencies changed)... 2024-02-10T06:49:35.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:35 smithi008 ceph-mon[24646]: Reconfiguring daemon prometheus.smithi008 on smithi008 2024-02-10T06:49:35.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 158846 12.08 MB/sec execute 176 sec latency 60.993 ms 2024-02-10T06:49:35.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:35 smithi029 ceph-mon[27077]: Reconfiguring daemon prometheus.smithi008 on smithi008 2024-02-10T06:49:36.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:36 smithi008 ceph-mon[24646]: pgmap v14: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 808 op/s 2024-02-10T06:49:36.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 159372 12.08 MB/sec execute 177 sec latency 38.381 ms 2024-02-10T06:49:36.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:36 smithi029 ceph-mon[27077]: pgmap v14: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 808 op/s 2024-02-10T06:49:37.556 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:49:37.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 159751 12.09 MB/sec execute 178 sec latency 35.194 ms 2024-02-10T06:49:38.546 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:38 smithi008 ceph-mon[24646]: pgmap v15: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.4 MiB/s wr, 551 op/s 2024-02-10T06:49:38.546 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:38 smithi008 ceph-mon[24646]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:49:38.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 160193 12.09 MB/sec execute 179 sec latency 33.887 ms 2024-02-10T06:49:38.692 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:49:38.692 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (117s) 15s ago 11m 16.7M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:49:38.692 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (11m) 15s ago 11m 6405k - 17.2.0 e1d6a67b021e 5fa2e5bdd7cc 2024-02-10T06:49:38.692 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (11m) 20s ago 11m 6405k - 17.2.0 e1d6a67b021e 7527ea77b5cc 2024-02-10T06:49:38.692 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (92s) 15s ago 11m 42.4M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:49:38.692 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (5m) 15s ago 5m 37.5M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:49:38.692 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (5m) 20s ago 5m 13.5M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:49:38.692 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (46s) 15s ago 13m 476M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:49:38.693 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283 running (2m) 20s ago 10m 78.9M - 19.0.0-1361-g290212a1 c328ab9dcfc7 754f1049995b 2024-02-10T06:49:38.693 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (13m) 15s ago 13m 55.0M 2048M 17.2.0 e1d6a67b021e 061312907e4b 2024-02-10T06:49:38.693 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (10m) 20s ago 10m 47.1M 2048M 17.2.0 e1d6a67b021e a790a61584bf 2024-02-10T06:49:38.693 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (5m) 15s ago 5m 280M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:49:38.693 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (5m) 20s ago 5m 83.7M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:49:38.693 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (87s) 15s ago 11m 11.3M - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:49:38.693 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (58s) 20s ago 10m 15.2M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:49:38.693 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (10m) 15s ago 10m 501M 1937M 17.2.0 e1d6a67b021e 139b7a92bfc4 2024-02-10T06:49:38.693 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (9m) 15s ago 9m 445M 1937M 17.2.0 e1d6a67b021e 0f1119e17a52 2024-02-10T06:49:38.693 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (9m) 15s ago 9m 337M 1937M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:49:38.693 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (8m) 15s ago 8m 477M 1937M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:49:38.693 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (8m) 20s ago 8m 495M 2705M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:49:38.693 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (7m) 20s ago 7m 570M 2705M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:49:38.693 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (7m) 20s ago 7m 482M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:49:38.693 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (7m) 20s ago 7m 370M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:49:38.694 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (61s) 15s ago 10m 53.4M - 2.33.4 514e6a882f6e 2901a03ac60a 2024-02-10T06:49:38.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:38 smithi029 ceph-mon[27077]: pgmap v15: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.4 MiB/s wr, 551 op/s 2024-02-10T06:49:38.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:38 smithi029 ceph-mon[27077]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:49:39.362 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:49:39.362 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:49:39.362 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:49:39.362 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:49:39.362 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:49:39.362 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:49:39.362 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:49:39.362 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:49:39.362 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:49:39.362 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:49:39.362 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:49:39.362 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:49:39.363 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:49:39.363 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:49:39.363 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-02-10T06:49:39.363 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:49:39.363 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:49:39.363 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:49:39.554 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:39 smithi008 ceph-mon[24646]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:49:39.554 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:39 smithi008 ceph-mon[24646]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:49:39.554 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:39 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/726548975' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:49:39.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 160708 12.08 MB/sec execute 180 sec latency 32.654 ms 2024-02-10T06:49:39.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:39 smithi029 ceph-mon[27077]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:49:39.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:39 smithi029 ceph-mon[27077]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:49:39.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:39 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/726548975' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:49:39.881 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:49:39.881 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:49:39.881 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:49:39.881 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:49:39.881 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:49:39.881 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-02-10T06:49:39.881 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:49:39.881 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "2/23 daemons upgraded", 2024-02-10T06:49:39.882 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-02-10T06:49:39.882 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:49:39.882 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:49:40.565 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:40 smithi008 ceph-mon[24646]: pgmap v16: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.4 MiB/s wr, 551 op/s 2024-02-10T06:49:40.577 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:49:40.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 161079 12.08 MB/sec execute 181 sec latency 42.034 ms 2024-02-10T06:49:40.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:40 smithi029 ceph-mon[27077]: pgmap v16: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.4 MiB/s wr, 551 op/s 2024-02-10T06:49:41.584 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:41 smithi008 ceph-mon[24646]: from='client.14882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:49:41.584 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:41 smithi008 ceph-mon[24646]: from='client.? 172.21.15.8:0/2793131161' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:49:41.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 161653 12.07 MB/sec execute 182 sec latency 63.221 ms 2024-02-10T06:49:41.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:41 smithi029 ceph-mon[27077]: from='client.14882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:49:41.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:41 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/2793131161' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:49:42.595 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:42 smithi008 ceph-mon[24646]: pgmap v17: 97 pgs: 97 active+clean; 351 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 813 op/s 2024-02-10T06:49:42.595 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:42 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:42.595 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:42 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:42.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 162079 12.07 MB/sec execute 183 sec latency 57.545 ms 2024-02-10T06:49:42.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:42 smithi029 ceph-mon[27077]: pgmap v17: 97 pgs: 97 active+clean; 351 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 813 op/s 2024-02-10T06:49:42.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:42 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:42.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:42 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:43.619 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:43 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:49:43.619 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:43 smithi008 ceph-mon[24646]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:49:43.619 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:43 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:49:43.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 162649 12.11 MB/sec execute 184 sec latency 40.458 ms 2024-02-10T06:49:43.656 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:43 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:49:43.656 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:43 smithi029 ceph-mon[27077]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:49:43.656 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:43 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:49:44.635 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:44 smithi008 ceph-mon[24646]: pgmap v18: 97 pgs: 97 active+clean; 351 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.1 MiB/s wr, 512 op/s 2024-02-10T06:49:44.635 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:44 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:49:44.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 163250 12.15 MB/sec execute 185 sec latency 37.970 ms 2024-02-10T06:49:44.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:44 smithi029 ceph-mon[27077]: pgmap v18: 97 pgs: 97 active+clean; 351 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.1 MiB/s wr, 512 op/s 2024-02-10T06:49:44.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:44 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:49:45.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 163568 12.14 MB/sec execute 186 sec latency 45.967 ms 2024-02-10T06:49:45.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:45 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi029.fuvrtf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:49:45.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:45 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:49:45.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:45 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:49:45.685 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:45 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi029.fuvrtf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:49:45.685 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:45 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:49:45.685 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:45 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:49:46.408 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:46 smithi008 ceph-mon[24646]: Upgrade: Updating mgr.smithi029.fuvrtf 2024-02-10T06:49:46.408 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:46 smithi008 ceph-mon[24646]: Deploying daemon mgr.smithi029.fuvrtf on smithi029 2024-02-10T06:49:46.408 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:46 smithi008 ceph-mon[24646]: pgmap v19: 97 pgs: 97 active+clean; 350 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 812 op/s 2024-02-10T06:49:46.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 163972 12.10 MB/sec execute 187 sec latency 38.018 ms 2024-02-10T06:49:46.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:46 smithi029 ceph-mon[27077]: Upgrade: Updating mgr.smithi029.fuvrtf 2024-02-10T06:49:46.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:46 smithi029 ceph-mon[27077]: Deploying daemon mgr.smithi029.fuvrtf on smithi029 2024-02-10T06:49:46.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:46 smithi029 ceph-mon[27077]: pgmap v19: 97 pgs: 97 active+clean; 350 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 812 op/s 2024-02-10T06:49:47.637 INFO:teuthology.orchestra.run.smithi008.stdout: 5 164236 12.07 MB/sec execute 188 sec latency 54.599 ms 2024-02-10T06:49:48.468 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:48 smithi008 ceph-mon[24646]: pgmap v20: 97 pgs: 97 active+clean; 350 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.8 MiB/s wr, 561 op/s 2024-02-10T06:49:48.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 164462 12.05 MB/sec execute 189 sec latency 73.210 ms 2024-02-10T06:49:48.637 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:48 smithi029 ceph-mon[27077]: pgmap v20: 97 pgs: 97 active+clean; 350 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.8 MiB/s wr, 561 op/s 2024-02-10T06:49:49.636 INFO:teuthology.orchestra.run.smithi008.stdout: 5 164730 12.03 MB/sec execute 190 sec latency 71.813 ms 2024-02-10T06:49:50.637 INFO:teuthology.orchestra.run.smithi008.stdout: 5 165029 12.00 MB/sec execute 191 sec latency 87.021 ms 2024-02-10T06:49:50.811 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:50 smithi008 ceph-mon[24646]: pgmap v21: 97 pgs: 97 active+clean; 350 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.8 MiB/s wr, 561 op/s 2024-02-10T06:49:50.811 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:50 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:50.811 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:50 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:50.811 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:50 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:49:50.876 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:50 smithi029 ceph-mon[27077]: pgmap v21: 97 pgs: 97 active+clean; 350 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.8 MiB/s wr, 561 op/s 2024-02-10T06:49:50.876 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:50 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:50.877 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:50 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:50.877 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:50 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:49:51.637 INFO:teuthology.orchestra.run.smithi008.stdout: 5 165389 11.97 MB/sec execute 192 sec latency 62.730 ms 2024-02-10T06:49:52.637 INFO:teuthology.orchestra.run.smithi008.stdout: 5 165697 11.94 MB/sec execute 193 sec latency 44.048 ms 2024-02-10T06:49:52.835 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:52 smithi008 ceph-mon[24646]: pgmap v22: 97 pgs: 97 active+clean; 344 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 776 op/s 2024-02-10T06:49:52.843 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:52 smithi029 ceph-mon[27077]: pgmap v22: 97 pgs: 97 active+clean; 344 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 776 op/s 2024-02-10T06:49:53.637 INFO:teuthology.orchestra.run.smithi008.stdout: 5 166173 11.98 MB/sec execute 194 sec latency 44.168 ms 2024-02-10T06:49:54.637 INFO:teuthology.orchestra.run.smithi008.stdout: 5 166632 11.98 MB/sec execute 195 sec latency 32.809 ms 2024-02-10T06:49:54.638 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:54 smithi008 ceph-mon[24646]: pgmap v23: 97 pgs: 97 active+clean; 344 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.2 MiB/s wr, 515 op/s 2024-02-10T06:49:54.639 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:54 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:54.639 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:54 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:54.946 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:54 smithi029 ceph-mon[27077]: pgmap v23: 97 pgs: 97 active+clean; 344 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.2 MiB/s wr, 515 op/s 2024-02-10T06:49:54.946 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:54 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:54.946 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:54 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:55.637 INFO:teuthology.orchestra.run.smithi008.stdout: 5 167049 11.99 MB/sec execute 196 sec latency 24.171 ms 2024-02-10T06:49:56.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:56 smithi029 ceph-mon[27077]: pgmap v24: 97 pgs: 97 active+clean; 346 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 724 op/s 2024-02-10T06:49:56.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:56 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:56.571 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:56 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:56.637 INFO:teuthology.orchestra.run.smithi008.stdout: 5 167550 11.97 MB/sec execute 197 sec latency 50.657 ms 2024-02-10T06:49:56.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:56 smithi008 ceph-mon[24646]: pgmap v24: 97 pgs: 97 active+clean; 346 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 724 op/s 2024-02-10T06:49:56.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:56 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:56.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:56 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:49:57.637 INFO:teuthology.orchestra.run.smithi008.stdout: 5 167878 11.95 MB/sec execute 198 sec latency 61.771 ms 2024-02-10T06:49:58.515 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:58 smithi008 ceph-mon[24646]: pgmap v25: 97 pgs: 97 active+clean; 346 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 424 op/s 2024-02-10T06:49:58.515 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:58 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:49:58.637 INFO:teuthology.orchestra.run.smithi008.stdout: 5 168249 11.97 MB/sec execute 199 sec latency 161.483 ms 2024-02-10T06:49:58.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:58 smithi029 ceph-mon[27077]: pgmap v25: 97 pgs: 97 active+clean; 346 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 424 op/s 2024-02-10T06:49:58.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:58 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:49:59.637 INFO:teuthology.orchestra.run.smithi008.stdout: 5 168771 11.96 MB/sec execute 200 sec latency 28.468 ms 2024-02-10T06:49:59.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:59 smithi008 ceph-mon[24646]: Standby manager daemon smithi029.fuvrtf restarted 2024-02-10T06:49:59.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:59 smithi008 ceph-mon[24646]: Standby manager daemon smithi029.fuvrtf started 2024-02-10T06:49:59.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:59 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.29:0/789800719' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/crt"}]: dispatch 2024-02-10T06:49:59.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:59 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.29:0/789800719' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:49:59.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:59 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.29:0/789800719' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/key"}]: dispatch 2024-02-10T06:49:59.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:49:59 smithi008 ceph-mon[24646]: from='mgr.? 172.21.15.29:0/789800719' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:49:59.872 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:59 smithi029 ceph-mon[27077]: Standby manager daemon smithi029.fuvrtf restarted 2024-02-10T06:49:59.872 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:59 smithi029 ceph-mon[27077]: Standby manager daemon smithi029.fuvrtf started 2024-02-10T06:49:59.872 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:59 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/789800719' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/crt"}]: dispatch 2024-02-10T06:49:59.872 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:59 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/789800719' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:49:59.873 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:59 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/789800719' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/key"}]: dispatch 2024-02-10T06:49:59.873 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:49:59 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/789800719' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:50:00.637 INFO:teuthology.orchestra.run.smithi008.stdout: 5 169077 11.92 MB/sec execute 201 sec latency 44.662 ms 2024-02-10T06:50:00.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: pgmap v26: 97 pgs: 97 active+clean; 346 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 424 op/s 2024-02-10T06:50:00.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: mgrmap e33: smithi008.pxifbt(active, since 47s), standbys: smithi029.fuvrtf 2024-02-10T06:50:00.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:00.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:00.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:00.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:00.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:00.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:50:00.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:00.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:00.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: overall HEALTH_OK 2024-02-10T06:50:00.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:50:00.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:50:00.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:00.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi008.pxifbt"}]: dispatch 2024-02-10T06:50:00.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi008.pxifbt"}]': finished 2024-02-10T06:50:00.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi029.fuvrtf"}]: dispatch 2024-02-10T06:50:00.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi029.fuvrtf"}]': finished 2024-02-10T06:50:00.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:00 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T06:50:00.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: pgmap v26: 97 pgs: 97 active+clean; 346 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 424 op/s 2024-02-10T06:50:00.776 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: mgrmap e33: smithi008.pxifbt(active, since 47s), standbys: smithi029.fuvrtf 2024-02-10T06:50:00.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:00.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:00.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:00.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:00.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:00.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:50:00.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:00.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:00.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: overall HEALTH_OK 2024-02-10T06:50:00.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:50:00.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:50:00.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:00.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi008.pxifbt"}]: dispatch 2024-02-10T06:50:00.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi008.pxifbt"}]': finished 2024-02-10T06:50:00.777 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi029.fuvrtf"}]: dispatch 2024-02-10T06:50:00.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi029.fuvrtf"}]': finished 2024-02-10T06:50:00.778 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:00 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T06:50:01.637 INFO:teuthology.orchestra.run.smithi008.stdout: 5 169333 11.90 MB/sec execute 202 sec latency 45.638 ms 2024-02-10T06:50:01.736 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:01 smithi008 ceph-mon[24646]: Upgrade: Setting container_image for all mgr 2024-02-10T06:50:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:01 smithi029 ceph-mon[27077]: Upgrade: Setting container_image for all mgr 2024-02-10T06:50:02.637 INFO:teuthology.orchestra.run.smithi008.stdout: 5 169712 11.92 MB/sec execute 203 sec latency 39.036 ms 2024-02-10T06:50:02.764 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:02 smithi008 ceph-mon[24646]: pgmap v27: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 690 op/s 2024-02-10T06:50:02.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:02 smithi029 ceph-mon[27077]: pgmap v27: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 690 op/s 2024-02-10T06:50:03.638 INFO:teuthology.orchestra.run.smithi008.stdout: 5 170064 11.90 MB/sec execute 204 sec latency 69.982 ms 2024-02-10T06:50:03.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:03 smithi008 ceph-mon[24646]: Upgrade: Updating mon.smithi008 2024-02-10T06:50:03.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:03 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:03.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:03 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:50:03.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:03 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:50:03.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:03 smithi008 ceph-mon[24646]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:03.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:03 smithi008 ceph-mon[24646]: Deploying daemon mon.smithi008 on smithi008 2024-02-10T06:50:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:03 smithi029 ceph-mon[27077]: Upgrade: Updating mon.smithi008 2024-02-10T06:50:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:03 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:03 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:50:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:03 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:50:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:03 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:03 smithi029 ceph-mon[27077]: Deploying daemon mon.smithi008 on smithi008 2024-02-10T06:50:04.638 INFO:teuthology.orchestra.run.smithi008.stdout: 5 170384 11.90 MB/sec execute 205 sec latency 35.719 ms 2024-02-10T06:50:04.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:04 smithi008 ceph-mon[24646]: pgmap v28: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 475 op/s 2024-02-10T06:50:04.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:04 smithi029 ceph-mon[27077]: pgmap v28: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 475 op/s 2024-02-10T06:50:05.568 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:05 smithi008 systemd[1]: Stopping Ceph mon.smithi008 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:50:05.638 INFO:teuthology.orchestra.run.smithi008.stdout: 5 170600 11.87 MB/sec execute 206 sec latency 54.595 ms 2024-02-10T06:50:05.816 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:05 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008[24642]: 2024-02-10T06:50:05.781+0000 7f9753c75700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi008 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:50:05.816 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:05 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008[24642]: 2024-02-10T06:50:05.781+0000 7f9753c75700 -1 mon.smithi008@0(leader) e2 *** Got Signal Terminated *** 2024-02-10T06:50:05.816 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:05 smithi008 podman[66251]: 2024-02-10 06:50:05.812371417 +0000 UTC m=+0.176672496 container died 061312907e4b95e3ed2a34007b7f6a69c9cb7067f0a83d4ba6fbd579d146331f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.openshift.expose-services=, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-10T06:50:06.638 INFO:teuthology.orchestra.run.smithi008.stdout: 5 170907 11.84 MB/sec execute 207 sec latency 78.407 ms 2024-02-10T06:50:06.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:06 smithi008 podman[66251]: 2024-02-10 06:50:06.665874387 +0000 UTC m=+1.030175455 container cleanup 061312907e4b95e3ed2a34007b7f6a69c9cb7067f0a83d4ba6fbd579d146331f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, RELEASE=HEAD, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, version=8, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, vcs-type=git, release=754) 2024-02-10T06:50:06.851 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:06 smithi008 bash[66251]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008 2024-02-10T06:50:07.115 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:07 smithi008 podman[66262]: 2024-02-10 06:50:07.087896891 +0000 UTC m=+1.275002767 container remove 061312907e4b95e3ed2a34007b7f6a69c9cb7067f0a83d4ba6fbd579d146331f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_CLEAN=True, distribution-scope=public, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, RELEASE=HEAD, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754) 2024-02-10T06:50:07.619 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:07 smithi008 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mon.smithi008.service: Deactivated successfully. 2024-02-10T06:50:07.619 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:07 smithi008 systemd[1]: Stopped Ceph mon.smithi008 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:50:07.619 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:07 smithi008 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mon.smithi008.service: Consumed 17.717s CPU time. 2024-02-10T06:50:07.638 INFO:teuthology.orchestra.run.smithi008.stdout: 5 171186 11.81 MB/sec execute 208 sec latency 90.083 ms 2024-02-10T06:50:07.885 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:07 smithi008 systemd[1]: Starting Ceph mon.smithi008 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:50:08.136 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:07 smithi008 podman[66396]: 2024-02-10 06:50:07.851158459 +0000 UTC m=+0.043669568 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:50:08.136 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 podman[66396]: 2024-02-10 06:50:08.054572632 +0000 UTC m=+0.247083722 container create ffaeb9e2c72103a9362c6dc13c899629eac8502f690ae3e5add304a1009dc0ed (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-290212a) 2024-02-10T06:50:08.638 INFO:teuthology.orchestra.run.smithi008.stdout: 5 171414 11.78 MB/sec execute 209 sec latency 46.437 ms 2024-02-10T06:50:08.651 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 podman[66396]: 2024-02-10 06:50:08.542924798 +0000 UTC m=+0.735435900 container init ffaeb9e2c72103a9362c6dc13c899629eac8502f690ae3e5add304a1009dc0ed (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-10T06:50:08.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 podman[66396]: 2024-02-10 06:50:08.54754753 +0000 UTC m=+0.740058628 container start ffaeb9e2c72103a9362c6dc13c899629eac8502f690ae3e5add304a1009dc0ed (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240131, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-02-10T06:50:08.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T06:50:08.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 2 2024-02-10T06:50:08.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: pidfile_write: ignore empty --pid-file 2024-02-10T06:50:08.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: load: jerasure load: lrc 2024-02-10T06:50:08.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: RocksDB version: 7.9.2 2024-02-10T06:50:08.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Git sha 0 2024-02-10T06:50:08.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T06:50:08.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: DB SUMMARY 2024-02-10T06:50:08.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: DB Session ID: F7Z1822QI9K7Q9V3DBS9 2024-02-10T06:50:08.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: CURRENT file: CURRENT 2024-02-10T06:50:08.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T06:50:08.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: MANIFEST file: MANIFEST-000015 size: 965 Bytes 2024-02-10T06:50:08.653 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi008/store.db dir, Total Num: 1, files: 000030.sst 2024-02-10T06:50:08.653 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi008/store.db: 000028.log size: 3024556 ; 2024-02-10T06:50:08.653 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.error_if_exists: 0 2024-02-10T06:50:08.653 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.create_if_missing: 0 2024-02-10T06:50:08.653 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.paranoid_checks: 1 2024-02-10T06:50:08.653 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T06:50:08.653 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T06:50:08.653 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T06:50:08.653 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.env: 0x5652e900aae0 2024-02-10T06:50:08.653 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T06:50:08.653 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.info_log: 0x5652e9df1760 2024-02-10T06:50:08.653 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T06:50:08.654 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.statistics: (nil) 2024-02-10T06:50:08.654 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.use_fsync: 0 2024-02-10T06:50:08.654 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_log_file_size: 0 2024-02-10T06:50:08.654 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T06:50:08.654 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T06:50:08.654 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T06:50:08.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T06:50:08.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.allow_fallocate: 1 2024-02-10T06:50:08.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T06:50:08.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T06:50:08.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.use_direct_reads: 0 2024-02-10T06:50:08.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T06:50:08.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T06:50:08.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.db_log_dir: 2024-02-10T06:50:08.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.wal_dir: 2024-02-10T06:50:08.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T06:50:08.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T06:50:08.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T06:50:08.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T06:50:08.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T06:50:08.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T06:50:08.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T06:50:08.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T06:50:08.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.write_buffer_manager: 0x5652e9dc5400 2024-02-10T06:50:08.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T06:50:08.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T06:50:08.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T06:50:08.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T06:50:08.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T06:50:08.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T06:50:08.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T06:50:08.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T06:50:08.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.unordered_write: 0 2024-02-10T06:50:08.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T06:50:08.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T06:50:08.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T06:50:08.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T06:50:08.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.row_cache: None 2024-02-10T06:50:08.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.wal_filter: None 2024-02-10T06:50:08.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T06:50:08.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T06:50:08.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.two_write_queues: 0 2024-02-10T06:50:08.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T06:50:08.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.wal_compression: 0 2024-02-10T06:50:08.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.atomic_flush: 0 2024-02-10T06:50:08.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T06:50:08.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T06:50:08.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T06:50:08.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.log_readahead_size: 0 2024-02-10T06:50:08.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T06:50:08.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T06:50:08.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T06:50:08.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T06:50:08.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T06:50:08.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T06:50:08.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T06:50:08.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_background_jobs: 2 2024-02-10T06:50:08.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_background_compactions: -1 2024-02-10T06:50:08.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_subcompactions: 1 2024-02-10T06:50:08.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T06:50:08.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T06:50:08.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T06:50:08.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T06:50:08.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T06:50:08.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T06:50:08.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T06:50:08.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T06:50:08.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_open_files: -1 2024-02-10T06:50:08.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T06:50:08.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T06:50:08.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T06:50:08.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T06:50:08.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_background_flushes: -1 2024-02-10T06:50:08.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Compression algorithms supported: 2024-02-10T06:50:08.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T06:50:08.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: kZSTD supported: 0 2024-02-10T06:50:08.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: kXpressCompression supported: 0 2024-02-10T06:50:08.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T06:50:08.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: kZlibCompression supported: 1 2024-02-10T06:50:08.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: kSnappyCompression supported: 1 2024-02-10T06:50:08.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: kLZ4Compression supported: 1 2024-02-10T06:50:08.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: kBZip2Compression supported: 0 2024-02-10T06:50:08.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T06:50:08.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T06:50:08.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi008/store.db/MANIFEST-000015 2024-02-10T06:50:08.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T06:50:08.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T06:50:08.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.merge_operator: 2024-02-10T06:50:08.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compaction_filter: None 2024-02-10T06:50:08.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T06:50:08.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T06:50:08.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T06:50:08.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T06:50:08.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5652e9df1880) 2024-02-10T06:50:08.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: cache_index_and_filter_blocks: 1 2024-02-10T06:50:08.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T06:50:08.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T06:50:08.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: pin_top_level_index_and_filter: 1 2024-02-10T06:50:08.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_type: 0 2024-02-10T06:50:08.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: data_block_index_type: 0 2024-02-10T06:50:08.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_shortening: 1 2024-02-10T06:50:08.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T06:50:08.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: checksum: 4 2024-02-10T06:50:08.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: no_block_cache: 0 2024-02-10T06:50:08.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache: 0x5652e9df5090 2024-02-10T06:50:08.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_name: BinnedLRUCache 2024-02-10T06:50:08.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_options: 2024-02-10T06:50:08.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: capacity : 536870912 2024-02-10T06:50:08.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: num_shard_bits : 4 2024-02-10T06:50:08.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: strict_capacity_limit : 0 2024-02-10T06:50:08.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: high_pri_pool_ratio: 0.000 2024-02-10T06:50:08.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_compressed: (nil) 2024-02-10T06:50:08.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: persistent_cache: (nil) 2024-02-10T06:50:08.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_size: 4096 2024-02-10T06:50:08.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_size_deviation: 10 2024-02-10T06:50:08.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_restart_interval: 16 2024-02-10T06:50:08.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_block_restart_interval: 1 2024-02-10T06:50:08.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: metadata_block_size: 4096 2024-02-10T06:50:08.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: partition_filters: 0 2024-02-10T06:50:08.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: use_delta_encoding: 1 2024-02-10T06:50:08.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: filter_policy: bloomfilter 2024-02-10T06:50:08.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: whole_key_filtering: 1 2024-02-10T06:50:08.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: verify_compression: 0 2024-02-10T06:50:08.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: read_amp_bytes_per_bit: 0 2024-02-10T06:50:08.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: format_version: 5 2024-02-10T06:50:08.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: enable_index_compression: 1 2024-02-10T06:50:08.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_align: 0 2024-02-10T06:50:08.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: max_auto_readahead_size: 262144 2024-02-10T06:50:08.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: prepopulate_block_cache: 0 2024-02-10T06:50:08.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: initial_auto_readahead_size: 8192 2024-02-10T06:50:08.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T06:50:08.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T06:50:08.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T06:50:08.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compression: NoCompression 2024-02-10T06:50:08.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T06:50:08.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T06:50:08.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T06:50:08.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.num_levels: 7 2024-02-10T06:50:08.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T06:50:08.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T06:50:08.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T06:50:08.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T06:50:08.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T06:50:08.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T06:50:08.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T06:50:08.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:50:08.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T06:50:08.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T06:50:08.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T06:50:08.665 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T06:50:08.665 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T06:50:08.665 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T06:50:08.665 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T06:50:08.665 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T06:50:08.665 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:50:08.665 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T06:50:08.665 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T06:50:08.665 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T06:50:08.665 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T06:50:08.665 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T06:50:08.665 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T06:50:08.665 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T06:50:08.665 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T06:50:08.666 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T06:50:08.666 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T06:50:08.666 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T06:50:08.666 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T06:50:08.666 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T06:50:08.666 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T06:50:08.667 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T06:50:08.667 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T06:50:08.667 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T06:50:08.667 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T06:50:08.667 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T06:50:08.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T06:50:08.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T06:50:08.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T06:50:08.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T06:50:08.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T06:50:08.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T06:50:08.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T06:50:08.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T06:50:08.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T06:50:08.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T06:50:08.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T06:50:08.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T06:50:08.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T06:50:08.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T06:50:08.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T06:50:08.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T06:50:08.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T06:50:08.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.table_properties_collectors: 2024-02-10T06:50:08.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.inplace_update_support: 0 2024-02-10T06:50:08.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T06:50:08.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T06:50:08.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T06:50:08.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T06:50:08.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.bloom_locality: 0 2024-02-10T06:50:08.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.max_successive_merges: 0 2024-02-10T06:50:08.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T06:50:08.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T06:50:08.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T06:50:08.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T06:50:08.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.ttl: 2592000 2024-02-10T06:50:08.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T06:50:08.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T06:50:08.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T06:50:08.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.enable_blob_files: false 2024-02-10T06:50:08.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.min_blob_size: 0 2024-02-10T06:50:08.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T06:50:08.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T06:50:08.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T06:50:08.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T06:50:08.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T06:50:08.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T06:50:08.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T06:50:08.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T06:50:08.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-02-10T06:50:08.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T06:50:08.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi008/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 11768, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T06:50:08.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-02-10T06:50:08.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0e262ca0-0a67-455c-94b9-bc71eda80113 2024-02-10T06:50:08.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707547808586626, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-02-10T06:50:08.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-02-10T06:50:08.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707547808599451, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2733925, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11769, "largest_seqno": 13317, "table_properties": {"data_size": 2726732, "index_size": 4510, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1669, "raw_key_size": 15130, "raw_average_key_size": 23, "raw_value_size": 2712550, "raw_average_value_size": 4205, "num_data_blocks": 206, "num_entries": 645, "num_filter_entries": 645, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707547808, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0e262ca0-0a67-455c-94b9-bc71eda80113", "db_session_id": "F7Z1822QI9K7Q9V3DBS9", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-02-10T06:50:08.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707547808599554, "job": 1, "event": "recovery_finished"} 2024-02-10T06:50:08.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-02-10T06:50:08.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T06:50:08.672 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi008/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T06:50:08.672 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5652e9ee4000 2024-02-10T06:50:08.672 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: DB pointer 0x5652e9ed0000 2024-02-10T06:50:08.672 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T06:50:08.672 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T06:50:08.672 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** DB Stats ** 2024-02-10T06:50:08.672 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:50:08.672 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T06:50:08.672 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T06:50:08.672 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:50:08.672 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T06:50:08.672 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T06:50:08.672 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:50:08.672 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-02-10T06:50:08.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-02-10T06:50:08.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T06:50:08.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T06:50:08.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: L0 1/0 2.61 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 418.0 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-02-10T06:50:08.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: L6 1/0 6.44 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-10T06:50:08.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Sum 2/0 9.04 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 418.0 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-02-10T06:50:08.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 418.0 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-02-10T06:50:08.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-02-10T06:50:08.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-02-10T06:50:08.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T06:50:08.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:50:08.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 418.0 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-02-10T06:50:08.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-02-10T06:50:08.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T06:50:08.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-02-10T06:50:08.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:50:08.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-02-10T06:50:08.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T06:50:08.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T06:50:08.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T06:50:08.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T06:50:08.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative compaction: 0.00 GB write, 164.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:50:08.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval compaction: 0.00 GB write, 164.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:50:08.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T06:50:08.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Block cache BinnedLRUCache@0x5652e9df5090#2 capacity: 512.00 MB usage: 6.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-02-10T06:50:08.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-02-10T06:50:08.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-02-10T06:50:08.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T06:50:08.674 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: starting mon.smithi008 rank 0 at public addrs [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] at bind addrs [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi008 fsid a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:50:08.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: mon.smithi008@-1(???) e2 preinit fsid a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:50:08.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: mon.smithi008@-1(???).mds e7 new map 2024-02-10T06:50:08.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: mon.smithi008@-1(???).mds e7 print_map 2024-02-10T06:50:08.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: e7 2024-02-10T06:50:08.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:50:08.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:50:08.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: legacy client fscid: 1 2024-02-10T06:50:08.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-02-10T06:50:08.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Filesystem 'foofs' (1) 2024-02-10T06:50:08.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: fs_name foofs 2024-02-10T06:50:08.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: epoch 5 2024-02-10T06:50:08.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-10T06:50:08.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: created 2024-02-10T06:44:02.111049+0000 2024-02-10T06:50:08.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: modified 2024-02-10T06:44:09.127989+0000 2024-02-10T06:50:08.675 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: tableserver 0 2024-02-10T06:50:08.676 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: root 0 2024-02-10T06:50:08.676 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: session_timeout 60 2024-02-10T06:50:08.676 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: session_autoclose 300 2024-02-10T06:50:08.676 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: max_file_size 1099511627776 2024-02-10T06:50:08.676 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: max_xattr_size 65536 2024-02-10T06:50:08.676 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: required_client_features {} 2024-02-10T06:50:08.676 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: last_failure 0 2024-02-10T06:50:08.676 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: last_failure_osd_epoch 0 2024-02-10T06:50:08.676 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:50:08.676 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: max_mds 1 2024-02-10T06:50:08.676 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: in 0 2024-02-10T06:50:08.676 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: up {0=14498} 2024-02-10T06:50:08.676 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: failed 2024-02-10T06:50:08.676 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: damaged 2024-02-10T06:50:08.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: stopped 2024-02-10T06:50:08.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: data_pools [3] 2024-02-10T06:50:08.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: metadata_pool 2 2024-02-10T06:50:08.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: inline_data disabled 2024-02-10T06:50:08.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: balancer 2024-02-10T06:50:08.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: bal_rank_mask -1 2024-02-10T06:50:08.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: standby_count_wanted 1 2024-02-10T06:50:08.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: [mds.foofs.smithi008.vfwcte{0:14498} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.8:6834/1608640408,v1:172.21.15.8:6835/1608640408] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:50:08.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-02-10T06:50:08.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-02-10T06:50:08.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Standby daemons: 2024-02-10T06:50:08.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-02-10T06:50:08.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: [mds.foofs.smithi029.bwmcyt{-1:24307} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.29:6832/1338152565,v1:172.21.15.29:6833/1338152565] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:50:08.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: mon.smithi008@-1(???).osd e72 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-10T06:50:08.678 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: mon.smithi008@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T06:50:08.678 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: mon.smithi008@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T06:50:08.678 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: mon.smithi008@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T06:50:08.678 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: mon.smithi008@-1(???).paxosservice(auth 1..29) refresh upgraded, format 0 -> 3 2024-02-10T06:50:08.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 bash[66396]: ffaeb9e2c72103a9362c6dc13c899629eac8502f690ae3e5add304a1009dc0ed 2024-02-10T06:50:08.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 systemd[1]: Started Ceph mon.smithi008 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:50:08.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:50:08.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:50:08.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: pgmap v30: 97 pgs: 97 active+clean; 345 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.8 MiB/s wr, 468 op/s 2024-02-10T06:50:08.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: mon.smithi008 calling monitor election 2024-02-10T06:50:08.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: mon.smithi008 is new leader, mons smithi008,smithi029 in quorum (ranks 0,1) 2024-02-10T06:50:08.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: monmap e2: 2 mons at {smithi008=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0],smithi029=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:50:08.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: fsmap foofs:1 {0=foofs.smithi008.vfwcte=up:active} 1 up:standby 2024-02-10T06:50:08.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T06:50:08.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: mgrmap e33: smithi008.pxifbt(active, since 56s), standbys: smithi029.fuvrtf 2024-02-10T06:50:08.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: from='mgr.14810 ' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:08.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: mgrmap e34: smithi008.pxifbt(active, since 56s), standbys: smithi029.fuvrtf 2024-02-10T06:50:08.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: overall HEALTH_OK 2024-02-10T06:50:08.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:08 smithi008 ceph-mon[66415]: from='mgr.14810 ' entity='' 2024-02-10T06:50:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:08 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:50:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:08 smithi029 ceph-mon[27077]: from='mgr.14810 172.21.15.8:0/4267340051' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:50:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:08 smithi029 ceph-mon[27077]: pgmap v30: 97 pgs: 97 active+clean; 345 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.8 MiB/s wr, 468 op/s 2024-02-10T06:50:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:08 smithi029 ceph-mon[27077]: mon.smithi008 calling monitor election 2024-02-10T06:50:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:08 smithi029 ceph-mon[27077]: mon.smithi008 is new leader, mons smithi008,smithi029 in quorum (ranks 0,1) 2024-02-10T06:50:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:08 smithi029 ceph-mon[27077]: monmap e2: 2 mons at {smithi008=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0],smithi029=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:50:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:08 smithi029 ceph-mon[27077]: fsmap foofs:1 {0=foofs.smithi008.vfwcte=up:active} 1 up:standby 2024-02-10T06:50:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:08 smithi029 ceph-mon[27077]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T06:50:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:08 smithi029 ceph-mon[27077]: mgrmap e33: smithi008.pxifbt(active, since 56s), standbys: smithi029.fuvrtf 2024-02-10T06:50:09.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:08 smithi029 ceph-mon[27077]: from='mgr.14810 ' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:09.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:08 smithi029 ceph-mon[27077]: mgrmap e34: smithi008.pxifbt(active, since 56s), standbys: smithi029.fuvrtf 2024-02-10T06:50:09.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:08 smithi029 ceph-mon[27077]: overall HEALTH_OK 2024-02-10T06:50:09.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:08 smithi029 ceph-mon[27077]: from='mgr.14810 ' entity='' 2024-02-10T06:50:09.638 INFO:teuthology.orchestra.run.smithi008.stdout: 5 171661 11.78 MB/sec execute 210 sec latency 63.149 ms 2024-02-10T06:50:10.638 INFO:teuthology.orchestra.run.smithi008.stdout: 5 171945 11.75 MB/sec execute 211 sec latency 74.686 ms 2024-02-10T06:50:10.958 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:10 smithi008 ceph-mon[66415]: mgrmap e35: smithi008.pxifbt(active, since 57s), standbys: smithi029.fuvrtf 2024-02-10T06:50:11.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:10 smithi029 ceph-mon[27077]: mgrmap e35: smithi008.pxifbt(active, since 57s), standbys: smithi029.fuvrtf 2024-02-10T06:50:11.638 INFO:teuthology.orchestra.run.smithi008.stdout: 5 172320 11.73 MB/sec execute 212 sec latency 54.286 ms 2024-02-10T06:50:12.638 INFO:teuthology.orchestra.run.smithi008.stdout: 5 172643 11.70 MB/sec execute 213 sec latency 41.215 ms 2024-02-10T06:50:13.639 INFO:teuthology.orchestra.run.smithi008.stdout: 5 172900 11.69 MB/sec execute 214 sec latency 50.430 ms 2024-02-10T06:50:14.638 INFO:teuthology.orchestra.run.smithi008.stdout: 5 173263 11.70 MB/sec execute 215 sec latency 46.639 ms 2024-02-10T06:50:15.638 INFO:teuthology.orchestra.run.smithi008.stdout: 5 173503 11.67 MB/sec execute 216 sec latency 78.098 ms 2024-02-10T06:50:16.638 INFO:teuthology.orchestra.run.smithi008.stdout: 5 173799 11.66 MB/sec execute 217 sec latency 40.452 ms 2024-02-10T06:50:17.453 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:17 smithi008 ceph-mon[66415]: Standby manager daemon smithi029.fuvrtf restarted 2024-02-10T06:50:17.453 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:17 smithi008 ceph-mon[66415]: Standby manager daemon smithi029.fuvrtf started 2024-02-10T06:50:17.453 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:17 smithi008 ceph-mon[66415]: from='mgr.? 172.21.15.29:0/1223576132' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/crt"}]: dispatch 2024-02-10T06:50:17.453 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:17 smithi008 ceph-mon[66415]: from='mgr.? 172.21.15.29:0/1223576132' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:50:17.453 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:17 smithi008 ceph-mon[66415]: from='mgr.? 172.21.15.29:0/1223576132' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/key"}]: dispatch 2024-02-10T06:50:17.453 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:17 smithi008 ceph-mon[66415]: from='mgr.? 172.21.15.29:0/1223576132' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:50:17.639 INFO:teuthology.orchestra.run.smithi008.stdout: 5 173981 11.63 MB/sec execute 218 sec latency 127.693 ms 2024-02-10T06:50:17.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:17 smithi029 ceph-mon[27077]: Standby manager daemon smithi029.fuvrtf restarted 2024-02-10T06:50:17.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:17 smithi029 ceph-mon[27077]: Standby manager daemon smithi029.fuvrtf started 2024-02-10T06:50:17.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:17 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/1223576132' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/crt"}]: dispatch 2024-02-10T06:50:17.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:17 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/1223576132' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:50:17.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:17 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/1223576132' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi029.fuvrtf/key"}]: dispatch 2024-02-10T06:50:17.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:17 smithi029 ceph-mon[27077]: from='mgr.? 172.21.15.29:0/1223576132' entity='mgr.smithi029.fuvrtf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:50:18.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: mgrmap e36: smithi008.pxifbt(active, since 64s), standbys: smithi029.fuvrtf 2024-02-10T06:50:18.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: Active manager daemon smithi008.pxifbt restarted 2024-02-10T06:50:18.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: Activating manager daemon smithi008.pxifbt 2024-02-10T06:50:18.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T06:50:18.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: mgrmap e37: smithi008.pxifbt(active, starting, since 0.00580779s), standbys: smithi029.fuvrtf 2024-02-10T06:50:18.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: failed to encode map e73 with expected crc 2024-02-10T06:50:18.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:50:18.484 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:50:18.484 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.vfwcte"}]: dispatch 2024-02-10T06:50:18.484 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi029.bwmcyt"}]: dispatch 2024-02-10T06:50:18.484 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: failed to encode map e73 with expected crc 2024-02-10T06:50:18.484 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr metadata", "who": "smithi008.pxifbt", "id": "smithi008.pxifbt"}]: dispatch 2024-02-10T06:50:18.484 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr metadata", "who": "smithi029.fuvrtf", "id": "smithi029.fuvrtf"}]: dispatch 2024-02-10T06:50:18.484 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:50:18.484 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:50:18.484 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:50:18.484 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:50:18.484 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:50:18.484 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:50:18.484 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:50:18.484 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:50:18.485 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:50:18.485 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:50:18.485 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:50:18.485 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: Manager daemon smithi008.pxifbt is now available 2024-02-10T06:50:18.485 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:50:18.485 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:50:18.485 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:50:18.485 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:50:18.485 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:50:18.487 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi029[27063]: 2024-02-10T06:50:18.207+0000 7f38122fa700 -1 mon.smithi029@1(peon).osd e73 update_from_paxos full map CRC mismatch, resetting to canonical 2024-02-10T06:50:18.487 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: mgrmap e36: smithi008.pxifbt(active, since 64s), standbys: smithi029.fuvrtf 2024-02-10T06:50:18.487 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: Active manager daemon smithi008.pxifbt restarted 2024-02-10T06:50:18.487 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: Activating manager daemon smithi008.pxifbt 2024-02-10T06:50:18.487 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T06:50:18.487 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: mgrmap e37: smithi008.pxifbt(active, starting, since 0.00580779s), standbys: smithi029.fuvrtf 2024-02-10T06:50:18.487 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: failed to encode map e73 with expected crc 2024-02-10T06:50:18.487 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:50:18.487 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:50:18.487 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.vfwcte"}]: dispatch 2024-02-10T06:50:18.487 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi029.bwmcyt"}]: dispatch 2024-02-10T06:50:18.487 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: failed to encode map e73 with expected crc 2024-02-10T06:50:18.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr metadata", "who": "smithi008.pxifbt", "id": "smithi008.pxifbt"}]: dispatch 2024-02-10T06:50:18.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr metadata", "who": "smithi029.fuvrtf", "id": "smithi029.fuvrtf"}]: dispatch 2024-02-10T06:50:18.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:50:18.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:50:18.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:50:18.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:50:18.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:50:18.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:50:18.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:50:18.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:50:18.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:50:18.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:50:18.488 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:50:18.489 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: Manager daemon smithi008.pxifbt is now available 2024-02-10T06:50:18.489 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:50:18.489 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:50:18.489 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:50:18.489 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:50:18.489 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:18 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.pxifbt/trash_purge_schedule"}]: dispatch 2024-02-10T06:50:18.638 INFO:teuthology.orchestra.run.smithi008.stdout: 5 174188 11.60 MB/sec execute 219 sec latency 64.892 ms 2024-02-10T06:50:19.249 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:50:19.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:19 smithi008 ceph-mon[66415]: failed to encode map e73 with expected crc 2024-02-10T06:50:19.498 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:19 smithi008 ceph-mon[66415]: mgrmap e38: smithi008.pxifbt(active, since 1.01158s), standbys: smithi029.fuvrtf 2024-02-10T06:50:19.639 INFO:teuthology.orchestra.run.smithi008.stdout: 5 174498 11.58 MB/sec execute 220 sec latency 68.109 ms 2024-02-10T06:50:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:19 smithi029 ceph-mon[27077]: failed to encode map e73 with expected crc 2024-02-10T06:50:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:19 smithi029 ceph-mon[27077]: mgrmap e38: smithi008.pxifbt(active, since 1.01158s), standbys: smithi029.fuvrtf 2024-02-10T06:50:20.406 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:50:20.406 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (2m) 20s ago 12m 16.7M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:50:20.406 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (12m) 20s ago 12m 6405k - 17.2.0 e1d6a67b021e 5fa2e5bdd7cc 2024-02-10T06:50:20.406 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (11m) 26s ago 11m 6405k - 17.2.0 e1d6a67b021e 7527ea77b5cc 2024-02-10T06:50:20.406 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (2m) 20s ago 12m 43.1M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:50:20.406 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (6m) 20s ago 6m 37.6M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:50:20.406 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (6m) 26s ago 6m 13.8M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:50:20.406 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (88s) 20s ago 13m 492M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:50:20.406 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (31s) 26s ago 11m 63.5M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:50:20.406 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (13m) 20s ago 13m 55.5M 2048M 17.2.0 e1d6a67b021e 061312907e4b 2024-02-10T06:50:20.406 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (11m) 26s ago 11m 45.9M 2048M 17.2.0 e1d6a67b021e a790a61584bf 2024-02-10T06:50:20.406 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (5m) 20s ago 5m 276M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:50:20.406 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (5m) 26s ago 5m 83.7M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:50:20.407 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (2m) 20s ago 12m 9017k - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:50:20.407 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (100s) 26s ago 11m 19.2M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:50:20.407 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (10m) 20s ago 10m 321M 1937M 17.2.0 e1d6a67b021e 139b7a92bfc4 2024-02-10T06:50:20.407 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (10m) 20s ago 10m 481M 1937M 17.2.0 e1d6a67b021e 0f1119e17a52 2024-02-10T06:50:20.407 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (9m) 20s ago 9m 374M 1937M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:50:20.407 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (9m) 20s ago 9m 506M 1937M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:50:20.407 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (9m) 26s ago 9m 533M 2705M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:50:20.407 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (8m) 26s ago 8m 337M 2705M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:50:20.407 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (8m) 26s ago 8m 532M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:50:20.407 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (7m) 26s ago 7m 438M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:50:20.407 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (38s) 20s ago 11m 48.9M - 2.33.4 514e6a882f6e 193819b86dea 2024-02-10T06:50:20.513 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:20 smithi008 ceph-mon[66415]: [10/Feb/2024:06:50:18] ENGINE Bus STARTING 2024-02-10T06:50:20.513 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:20 smithi008 ceph-mon[66415]: [10/Feb/2024:06:50:19] ENGINE Serving on http://172.21.15.8:8765 2024-02-10T06:50:20.513 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:20 smithi008 ceph-mon[66415]: from='client.34108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:50:20.513 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:20 smithi008 ceph-mon[66415]: [10/Feb/2024:06:50:19] ENGINE Serving on https://172.21.15.8:7150 2024-02-10T06:50:20.513 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:20 smithi008 ceph-mon[66415]: [10/Feb/2024:06:50:19] ENGINE Bus STARTED 2024-02-10T06:50:20.513 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:20 smithi008 ceph-mon[66415]: pgmap v3: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-10T06:50:20.513 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:20 smithi008 ceph-mon[66415]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:50:20.631 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:20 smithi029 ceph-mon[27077]: [10/Feb/2024:06:50:18] ENGINE Bus STARTING 2024-02-10T06:50:20.632 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:20 smithi029 ceph-mon[27077]: [10/Feb/2024:06:50:19] ENGINE Serving on http://172.21.15.8:8765 2024-02-10T06:50:20.632 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:20 smithi029 ceph-mon[27077]: from='client.34108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:50:20.632 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:20 smithi029 ceph-mon[27077]: [10/Feb/2024:06:50:19] ENGINE Serving on https://172.21.15.8:7150 2024-02-10T06:50:20.632 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:20 smithi029 ceph-mon[27077]: [10/Feb/2024:06:50:19] ENGINE Bus STARTED 2024-02-10T06:50:20.632 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:20 smithi029 ceph-mon[27077]: pgmap v3: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-10T06:50:20.632 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:20 smithi029 ceph-mon[27077]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:50:20.639 INFO:teuthology.orchestra.run.smithi008.stdout: 5 174774 11.55 MB/sec execute 221 sec latency 68.731 ms 2024-02-10T06:50:21.080 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:50:21.080 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:50:21.080 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T06:50:21.080 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T06:50:21.080 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:50:21.081 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:50:21.081 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:50:21.081 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:50:21.081 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:50:21.081 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:50:21.081 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:50:21.081 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:50:21.081 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:50:21.081 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:50:21.081 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:50:21.081 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-02-10T06:50:21.081 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:50:21.081 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:50:21.081 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:50:21.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:21 smithi008 ceph-mon[66415]: pgmap v4: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-10T06:50:21.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:21 smithi008 ceph-mon[66415]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:50:21.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:21 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/3709386471' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:50:21.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:21 smithi008 ceph-mon[66415]: mgrmap e39: smithi008.pxifbt(active, since 3s), standbys: smithi029.fuvrtf 2024-02-10T06:50:21.639 INFO:teuthology.orchestra.run.smithi008.stdout: 5 175020 11.53 MB/sec execute 222 sec latency 54.728 ms 2024-02-10T06:50:21.652 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:50:21.652 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:50:21.652 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:50:21.652 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:50:21.652 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:50:21.652 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-02-10T06:50:21.652 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:50:21.652 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "2/23 daemons upgraded", 2024-02-10T06:50:21.652 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-02-10T06:50:21.653 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:50:21.653 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:50:21.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:21 smithi029 ceph-mon[27077]: pgmap v4: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-10T06:50:21.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:21 smithi029 ceph-mon[27077]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:50:21.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:21 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/3709386471' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:50:21.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:21 smithi029 ceph-mon[27077]: mgrmap e39: smithi008.pxifbt(active, since 3s), standbys: smithi029.fuvrtf 2024-02-10T06:50:22.327 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:50:22.561 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:22 smithi008 ceph-mon[66415]: from='client.24667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:50:22.561 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:22 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/3519996820' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:50:22.639 INFO:teuthology.orchestra.run.smithi008.stdout: 5 175305 11.52 MB/sec execute 223 sec latency 47.112 ms 2024-02-10T06:50:22.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:22 smithi029 ceph-mon[27077]: from='client.24667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:50:22.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:22 smithi029 ceph-mon[27077]: from='client.? 172.21.15.8:0/3519996820' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:50:23.578 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:23 smithi008 ceph-mon[66415]: pgmap v5: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-10T06:50:23.578 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:23 smithi008 ceph-mon[66415]: mgrmap e40: smithi008.pxifbt(active, since 4s), standbys: smithi029.fuvrtf 2024-02-10T06:50:23.578 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:23 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:23.578 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:23 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:23.639 INFO:teuthology.orchestra.run.smithi008.stdout: 5 175686 11.51 MB/sec execute 224 sec latency 48.684 ms 2024-02-10T06:50:23.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:23 smithi029 ceph-mon[27077]: pgmap v5: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-10T06:50:23.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:23 smithi029 ceph-mon[27077]: mgrmap e40: smithi008.pxifbt(active, since 4s), standbys: smithi029.fuvrtf 2024-02-10T06:50:23.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:23 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:23.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:23 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:24.639 INFO:teuthology.orchestra.run.smithi008.stdout: 5 176050 11.49 MB/sec execute 225 sec latency 43.996 ms 2024-02-10T06:50:25.100 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:24 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:25.100 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:24 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:25.100 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:24 smithi008 ceph-mon[66415]: pgmap v6: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 41 KiB/s rd, 1.8 MiB/s wr, 204 op/s 2024-02-10T06:50:25.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:24 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:24 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:24 smithi029 ceph-mon[27077]: pgmap v6: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 41 KiB/s rd, 1.8 MiB/s wr, 204 op/s 2024-02-10T06:50:25.639 INFO:teuthology.orchestra.run.smithi008.stdout: 5 176341 11.47 MB/sec execute 226 sec latency 38.826 ms 2024-02-10T06:50:26.640 INFO:teuthology.orchestra.run.smithi008.stdout: 5 176647 11.47 MB/sec execute 227 sec latency 53.716 ms 2024-02-10T06:50:27.391 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:27 smithi008 ceph-mon[66415]: pgmap v7: 97 pgs: 97 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 2.8 MiB/s wr, 339 op/s 2024-02-10T06:50:27.558 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:27 smithi029 ceph-mon[27077]: pgmap v7: 97 pgs: 97 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 2.8 MiB/s wr, 339 op/s 2024-02-10T06:50:27.640 INFO:teuthology.orchestra.run.smithi008.stdout: 5 176965 11.47 MB/sec execute 228 sec latency 50.611 ms 2024-02-10T06:50:28.640 INFO:teuthology.orchestra.run.smithi008.stdout: 5 177295 11.46 MB/sec execute 229 sec latency 48.490 ms 2024-02-10T06:50:28.917 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:28 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:28.917 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:28 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:28.917 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:28 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:28.917 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:28 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:28.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:28 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:28.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:28 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:28.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:28 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:28.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:28 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:29.640 INFO:teuthology.orchestra.run.smithi008.stdout: 5 177532 11.44 MB/sec execute 230 sec latency 57.196 ms 2024-02-10T06:50:29.956 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:29 smithi008 ceph-mon[66415]: pgmap v8: 97 pgs: 97 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 2.2 MiB/s wr, 263 op/s 2024-02-10T06:50:29.956 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:29 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:29.956 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:29 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:29.956 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:29 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:29.956 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:29 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:50:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:29 smithi029 ceph-mon[27077]: pgmap v8: 97 pgs: 97 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 2.2 MiB/s wr, 263 op/s 2024-02-10T06:50:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:29 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:29 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:29 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:29 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:50:30.640 INFO:teuthology.orchestra.run.smithi008.stdout: 5 177876 11.43 MB/sec execute 231 sec latency 64.790 ms 2024-02-10T06:50:30.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:30 smithi008 ceph-mon[66415]: Updating smithi008:/etc/ceph/ceph.conf 2024-02-10T06:50:30.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:30 smithi008 ceph-mon[66415]: Updating smithi029:/etc/ceph/ceph.conf 2024-02-10T06:50:30.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:30 smithi008 ceph-mon[66415]: Updating smithi029:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:50:30.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:30 smithi008 ceph-mon[66415]: Updating smithi008:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:50:30.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:30 smithi008 ceph-mon[66415]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:50:30.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:30 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:30.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:30 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:30.962 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:30 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:30.962 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:30 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:30.962 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:30 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:30.962 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:30 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:30 smithi029 ceph-mon[27077]: Updating smithi008:/etc/ceph/ceph.conf 2024-02-10T06:50:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:30 smithi029 ceph-mon[27077]: Updating smithi029:/etc/ceph/ceph.conf 2024-02-10T06:50:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:30 smithi029 ceph-mon[27077]: Updating smithi029:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:50:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:30 smithi029 ceph-mon[27077]: Updating smithi008:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:50:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:30 smithi029 ceph-mon[27077]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:50:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:30 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:30 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:30 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:30 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:30 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:30 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:31.640 INFO:teuthology.orchestra.run.smithi008.stdout: 5 178318 11.41 MB/sec execute 232 sec latency 45.181 ms 2024-02-10T06:50:31.974 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:31 smithi008 ceph-mon[66415]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:50:31.974 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:31 smithi008 ceph-mon[66415]: Updating smithi029:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:50:31.974 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:31 smithi008 ceph-mon[66415]: pgmap v9: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 4.1 MiB/s wr, 423 op/s 2024-02-10T06:50:31.974 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:31 smithi008 ceph-mon[66415]: Updating smithi008:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:50:31.974 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:50:31.974 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:50:31.974 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T06:50:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:31 smithi029 ceph-mon[27077]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:50:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:31 smithi029 ceph-mon[27077]: Updating smithi029:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:50:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:31 smithi029 ceph-mon[27077]: pgmap v9: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 4.1 MiB/s wr, 423 op/s 2024-02-10T06:50:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:31 smithi029 ceph-mon[27077]: Updating smithi008:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:50:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:31 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:50:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:31 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:50:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:31 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T06:50:32.640 INFO:teuthology.orchestra.run.smithi008.stdout: 5 178734 11.43 MB/sec execute 233 sec latency 45.838 ms 2024-02-10T06:50:33.640 INFO:teuthology.orchestra.run.smithi008.stdout: 5 179156 11.43 MB/sec execute 234 sec latency 58.568 ms 2024-02-10T06:50:33.755 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:33 smithi008 ceph-mon[66415]: pgmap v10: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 3.7 MiB/s wr, 388 op/s 2024-02-10T06:50:33.755 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:33 smithi008 ceph-mon[66415]: Upgrade: Updating mon.smithi029 2024-02-10T06:50:33.755 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:33 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:33.756 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:33 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:50:33.756 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:33 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:50:33.756 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:33 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:33.756 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:33 smithi008 ceph-mon[66415]: Deploying daemon mon.smithi029 on smithi029 2024-02-10T06:50:33.756 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:33 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:50:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:33 smithi029 ceph-mon[27077]: pgmap v10: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 3.7 MiB/s wr, 388 op/s 2024-02-10T06:50:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:33 smithi029 ceph-mon[27077]: Upgrade: Updating mon.smithi029 2024-02-10T06:50:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:33 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:33 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:50:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:33 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:50:33.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:33 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:33.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:33 smithi029 ceph-mon[27077]: Deploying daemon mon.smithi029 on smithi029 2024-02-10T06:50:33.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:33 smithi029 ceph-mon[27077]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:50:34.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 179622 11.41 MB/sec execute 235 sec latency 42.513 ms 2024-02-10T06:50:35.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 179923 11.40 MB/sec execute 236 sec latency 128.197 ms 2024-02-10T06:50:35.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:35 smithi008 ceph-mon[66415]: pgmap v11: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 5.4 MiB/s wr, 587 op/s 2024-02-10T06:50:35.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:35 smithi029 ceph-mon[27077]: pgmap v11: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 5.4 MiB/s wr, 587 op/s 2024-02-10T06:50:36.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:36 smithi029 systemd[1]: Stopping Ceph mon.smithi029 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:50:36.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 180410 11.42 MB/sec execute 237 sec latency 43.863 ms 2024-02-10T06:50:36.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:36 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi029[27063]: 2024-02-10T06:50:36.704+0000 7f3817b05700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi029 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:50:36.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:36 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi029[27063]: 2024-02-10T06:50:36.704+0000 7f3817b05700 -1 mon.smithi029@1(peon) e2 *** Got Signal Terminated *** 2024-02-10T06:50:36.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:36 smithi029 podman[53043]: 2024-02-10 06:50:36.766229889 +0000 UTC m=+0.460580589 container died a790a61584bfb9527cac4501975fbd9af046e5da883706ed4b1c7f2eae2fa2ef (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi029, vcs-type=git, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, architecture=x86_64, name=centos-stream, GIT_BRANCH=HEAD, distribution-scope=public, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) 2024-02-10T06:50:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:37 smithi029 podman[53043]: 2024-02-10 06:50:37.18296437 +0000 UTC m=+0.877315065 container cleanup a790a61584bfb9527cac4501975fbd9af046e5da883706ed4b1c7f2eae2fa2ef (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi029, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vcs-type=git, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, GIT_CLEAN=True, version=8, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , RELEASE=HEAD) 2024-02-10T06:50:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:37 smithi029 bash[53043]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi029 2024-02-10T06:50:37.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 180860 11.43 MB/sec execute 238 sec latency 37.574 ms 2024-02-10T06:50:37.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:37 smithi029 podman[53057]: 2024-02-10 06:50:37.438317677 +0000 UTC m=+0.671137673 container remove a790a61584bfb9527cac4501975fbd9af046e5da883706ed4b1c7f2eae2fa2ef (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi029, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vcs-type=git, release=754, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, ceph=True) 2024-02-10T06:50:37.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:37 smithi029 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mon.smithi029.service: Deactivated successfully. 2024-02-10T06:50:37.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:37 smithi029 systemd[1]: Stopped Ceph mon.smithi029 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:50:37.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:37 smithi029 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mon.smithi029.service: Consumed 7.305s CPU time. 2024-02-10T06:50:38.048 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:37 smithi029 systemd[1]: Starting Ceph mon.smithi029 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:50:38.049 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 podman[53191]: 2024-02-10 06:50:37.920480164 +0000 UTC m=+0.019188522 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:50:38.303 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 podman[53191]: 2024-02-10 06:50:38.049012463 +0000 UTC m=+0.147720813 container create 9492552149b2e4b58d8a07a94a69adfc7dce601716d19403d34b90b61a5a14ba (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi029, GIT_BRANCH=HEAD, RELEASE=main-290212a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-10T06:50:38.304 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 podman[53191]: 2024-02-10 06:50:38.297638113 +0000 UTC m=+0.396346463 container init 9492552149b2e4b58d8a07a94a69adfc7dce601716d19403d34b90b61a5a14ba (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi029, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-10T06:50:38.304 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 podman[53191]: 2024-02-10 06:50:38.301203825 +0000 UTC m=+0.399912177 container start 9492552149b2e4b58d8a07a94a69adfc7dce601716d19403d34b90b61a5a14ba (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi029, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, io.buildah.version=1.33.2) 2024-02-10T06:50:38.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 181227 11.43 MB/sec execute 239 sec latency 47.299 ms 2024-02-10T06:50:38.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T06:50:38.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 2 2024-02-10T06:50:38.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: pidfile_write: ignore empty --pid-file 2024-02-10T06:50:38.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: load: jerasure load: lrc 2024-02-10T06:50:38.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: RocksDB version: 7.9.2 2024-02-10T06:50:38.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Git sha 0 2024-02-10T06:50:38.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T06:50:38.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: DB SUMMARY 2024-02-10T06:50:38.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: DB Session ID: WLL64SI9LJZ3TRG0WJ3Z 2024-02-10T06:50:38.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: CURRENT file: CURRENT 2024-02-10T06:50:38.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T06:50:38.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: MANIFEST file: MANIFEST-000009 size: 890 Bytes 2024-02-10T06:50:38.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi029/store.db dir, Total Num: 1, files: 000024.sst 2024-02-10T06:50:38.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi029/store.db: 000022.log size: 7781668 ; 2024-02-10T06:50:38.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.error_if_exists: 0 2024-02-10T06:50:38.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.create_if_missing: 0 2024-02-10T06:50:38.709 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.paranoid_checks: 1 2024-02-10T06:50:38.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T06:50:38.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T06:50:38.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T06:50:38.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.env: 0x565067620ae0 2024-02-10T06:50:38.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T06:50:38.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.info_log: 0x56506811d760 2024-02-10T06:50:38.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T06:50:38.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.statistics: (nil) 2024-02-10T06:50:38.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.use_fsync: 0 2024-02-10T06:50:38.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_log_file_size: 0 2024-02-10T06:50:38.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T06:50:38.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T06:50:38.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T06:50:38.710 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T06:50:38.711 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.allow_fallocate: 1 2024-02-10T06:50:38.711 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T06:50:38.711 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T06:50:38.711 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.use_direct_reads: 0 2024-02-10T06:50:38.711 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T06:50:38.711 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T06:50:38.711 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.db_log_dir: 2024-02-10T06:50:38.711 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.wal_dir: 2024-02-10T06:50:38.711 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T06:50:38.711 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T06:50:38.711 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T06:50:38.711 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T06:50:38.711 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T06:50:38.711 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T06:50:38.711 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T06:50:38.711 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T06:50:38.712 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.write_buffer_manager: 0x5650680f1400 2024-02-10T06:50:38.712 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T06:50:38.712 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T06:50:38.712 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T06:50:38.712 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T06:50:38.712 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T06:50:38.712 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T06:50:38.712 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T06:50:38.712 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T06:50:38.712 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.unordered_write: 0 2024-02-10T06:50:38.712 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T06:50:38.712 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T06:50:38.712 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T06:50:38.712 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T06:50:38.712 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.row_cache: None 2024-02-10T06:50:38.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.wal_filter: None 2024-02-10T06:50:38.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T06:50:38.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T06:50:38.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.two_write_queues: 0 2024-02-10T06:50:38.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T06:50:38.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.wal_compression: 0 2024-02-10T06:50:38.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.atomic_flush: 0 2024-02-10T06:50:38.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T06:50:38.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T06:50:38.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T06:50:38.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.log_readahead_size: 0 2024-02-10T06:50:38.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T06:50:38.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T06:50:38.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T06:50:38.713 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T06:50:38.714 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T06:50:38.714 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T06:50:38.715 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T06:50:38.715 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_background_jobs: 2 2024-02-10T06:50:38.715 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_background_compactions: -1 2024-02-10T06:50:38.715 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_subcompactions: 1 2024-02-10T06:50:38.715 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T06:50:38.715 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T06:50:38.715 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T06:50:38.715 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T06:50:38.715 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T06:50:38.715 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T06:50:38.715 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T06:50:38.715 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T06:50:38.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_open_files: -1 2024-02-10T06:50:38.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T06:50:38.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T06:50:38.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T06:50:38.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T06:50:38.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_background_flushes: -1 2024-02-10T06:50:38.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Compression algorithms supported: 2024-02-10T06:50:38.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T06:50:38.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: kZSTD supported: 0 2024-02-10T06:50:38.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: kXpressCompression supported: 0 2024-02-10T06:50:38.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T06:50:38.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: kZlibCompression supported: 1 2024-02-10T06:50:38.716 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: kSnappyCompression supported: 1 2024-02-10T06:50:38.717 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: kLZ4Compression supported: 1 2024-02-10T06:50:38.717 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: kBZip2Compression supported: 0 2024-02-10T06:50:38.717 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T06:50:38.717 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T06:50:38.717 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi029/store.db/MANIFEST-000009 2024-02-10T06:50:38.717 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T06:50:38.717 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T06:50:38.717 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.merge_operator: 2024-02-10T06:50:38.717 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compaction_filter: None 2024-02-10T06:50:38.717 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T06:50:38.717 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T06:50:38.717 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T06:50:38.717 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T06:50:38.717 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56506811d880) 2024-02-10T06:50:38.717 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: cache_index_and_filter_blocks: 1 2024-02-10T06:50:38.718 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T06:50:38.718 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T06:50:38.718 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: pin_top_level_index_and_filter: 1 2024-02-10T06:50:38.718 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: index_type: 0 2024-02-10T06:50:38.718 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: data_block_index_type: 0 2024-02-10T06:50:38.718 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: index_shortening: 1 2024-02-10T06:50:38.718 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T06:50:38.718 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: checksum: 4 2024-02-10T06:50:38.718 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: no_block_cache: 0 2024-02-10T06:50:38.718 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_cache: 0x565068121090 2024-02-10T06:50:38.718 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_cache_name: BinnedLRUCache 2024-02-10T06:50:38.718 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_cache_options: 2024-02-10T06:50:38.718 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: capacity : 536870912 2024-02-10T06:50:38.718 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: num_shard_bits : 4 2024-02-10T06:50:38.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: strict_capacity_limit : 0 2024-02-10T06:50:38.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: high_pri_pool_ratio: 0.000 2024-02-10T06:50:38.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_cache_compressed: (nil) 2024-02-10T06:50:38.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: persistent_cache: (nil) 2024-02-10T06:50:38.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_size: 4096 2024-02-10T06:50:38.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_size_deviation: 10 2024-02-10T06:50:38.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_restart_interval: 16 2024-02-10T06:50:38.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: index_block_restart_interval: 1 2024-02-10T06:50:38.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: metadata_block_size: 4096 2024-02-10T06:50:38.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: partition_filters: 0 2024-02-10T06:50:38.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: use_delta_encoding: 1 2024-02-10T06:50:38.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: filter_policy: bloomfilter 2024-02-10T06:50:38.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: whole_key_filtering: 1 2024-02-10T06:50:38.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: verify_compression: 0 2024-02-10T06:50:38.719 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: read_amp_bytes_per_bit: 0 2024-02-10T06:50:38.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: format_version: 5 2024-02-10T06:50:38.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: enable_index_compression: 1 2024-02-10T06:50:38.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: block_align: 0 2024-02-10T06:50:38.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: max_auto_readahead_size: 262144 2024-02-10T06:50:38.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: prepopulate_block_cache: 0 2024-02-10T06:50:38.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: initial_auto_readahead_size: 8192 2024-02-10T06:50:38.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T06:50:38.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T06:50:38.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T06:50:38.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compression: NoCompression 2024-02-10T06:50:38.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T06:50:38.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T06:50:38.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T06:50:38.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.num_levels: 7 2024-02-10T06:50:38.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T06:50:38.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T06:50:38.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T06:50:38.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T06:50:38.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T06:50:38.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T06:50:38.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T06:50:38.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:50:38.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T06:50:38.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T06:50:38.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T06:50:38.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T06:50:38.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T06:50:38.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T06:50:38.721 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T06:50:38.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T06:50:38.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:50:38.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T06:50:38.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T06:50:38.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T06:50:38.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T06:50:38.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T06:50:38.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T06:50:38.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T06:50:38.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T06:50:38.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T06:50:38.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T06:50:38.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T06:50:38.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T06:50:38.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T06:50:38.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T06:50:38.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T06:50:38.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T06:50:38.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T06:50:38.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T06:50:38.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T06:50:38.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T06:50:38.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T06:50:38.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T06:50:38.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T06:50:38.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T06:50:38.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T06:50:38.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T06:50:38.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T06:50:38.724 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T06:50:38.724 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T06:50:38.724 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T06:50:38.724 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T06:50:38.724 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T06:50:38.724 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T06:50:38.724 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T06:50:38.724 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T06:50:38.724 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T06:50:38.724 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.table_properties_collectors: 2024-02-10T06:50:38.724 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.inplace_update_support: 0 2024-02-10T06:50:38.724 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T06:50:38.724 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T06:50:38.724 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T06:50:38.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T06:50:38.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.bloom_locality: 0 2024-02-10T06:50:38.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.max_successive_merges: 0 2024-02-10T06:50:38.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T06:50:38.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T06:50:38.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T06:50:38.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T06:50:38.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.ttl: 2592000 2024-02-10T06:50:38.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T06:50:38.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T06:50:38.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T06:50:38.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.enable_blob_files: false 2024-02-10T06:50:38.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.min_blob_size: 0 2024-02-10T06:50:38.725 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T06:50:38.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T06:50:38.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T06:50:38.727 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T06:50:38.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T06:50:38.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T06:50:38.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T06:50:38.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T06:50:38.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 24.sst 2024-02-10T06:50:38.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T06:50:38.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi029/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 26, last_sequence is 12150, log_number is 22,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T06:50:38.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 22 2024-02-10T06:50:38.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7ef803a5-359d-469f-9adc-4cf39393ee36 2024-02-10T06:50:38.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707547838336123, "job": 1, "event": "recovery_started", "wal_files": [22]} 2024-02-10T06:50:38.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #22 mode 2 2024-02-10T06:50:38.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707547838355311, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 27, "file_size": 4687571, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12155, "largest_seqno": 14488, "table_properties": {"data_size": 4678073, "index_size": 6238, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2245, "raw_key_size": 20444, "raw_average_key_size": 23, "raw_value_size": 4658965, "raw_average_value_size": 5306, "num_data_blocks": 283, "num_entries": 878, "num_filter_entries": 878, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707547838, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7ef803a5-359d-469f-9adc-4cf39393ee36", "db_session_id": "WLL64SI9LJZ3TRG0WJ3Z", "orig_file_number": 27, "seqno_to_time_mapping": "N/A"}} 2024-02-10T06:50:38.728 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707547838355423, "job": 1, "event": "recovery_finished"} 2024-02-10T06:50:38.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: [db/version_set.cc:5047] Creating manifest 29 2024-02-10T06:50:38.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T06:50:38.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi029/store.db/000022.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T06:50:38.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x565068210000 2024-02-10T06:50:38.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: DB pointer 0x5650681fc000 2024-02-10T06:50:38.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T06:50:38.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T06:50:38.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ** DB Stats ** 2024-02-10T06:50:38.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:50:38.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T06:50:38.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T06:50:38.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:50:38.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T06:50:38.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T06:50:38.729 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:50:38.730 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-02-10T06:50:38.730 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ** Compaction Stats [default] ** 2024-02-10T06:50:38.730 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T06:50:38.730 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T06:50:38.730 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: L0 1/0 4.47 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 657.6 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-02-10T06:50:38.730 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: L6 1/0 6.44 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-10T06:50:38.730 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Sum 2/0 10.91 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 657.6 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-02-10T06:50:38.730 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 657.6 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-02-10T06:50:38.730 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-02-10T06:50:38.730 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ** Compaction Stats [default] ** 2024-02-10T06:50:38.730 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T06:50:38.730 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:50:38.730 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 657.6 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-02-10T06:50:38.730 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-02-10T06:50:38.730 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T06:50:38.731 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-02-10T06:50:38.731 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:50:38.731 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-02-10T06:50:38.731 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T06:50:38.731 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T06:50:38.731 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T06:50:38.731 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T06:50:38.731 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Cumulative compaction: 0.00 GB write, 199.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:50:38.731 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Interval compaction: 0.00 GB write, 199.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:50:38.731 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T06:50:38.731 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Block cache BinnedLRUCache@0x565068121090#2 capacity: 512.00 MB usage: 8.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-02-10T06:50:38.731 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.28 KB,0.000435114%) IndexBlock(1,6.58 KB,0.00125468%) Misc(1,0.00 KB,0%) 2024-02-10T06:50:38.731 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-02-10T06:50:38.731 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T06:50:38.732 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: starting mon.smithi029 rank 1 at public addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] at bind addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi029 fsid a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:50:38.732 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: mon.smithi029@-1(???) e2 preinit fsid a5f941ec-c7de-11ee-95b8-87774f69a715 2024-02-10T06:50:38.732 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: mon.smithi029@-1(???).mds e7 new map 2024-02-10T06:50:38.732 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: mon.smithi029@-1(???).mds e7 print_map 2024-02-10T06:50:38.732 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: e7 2024-02-10T06:50:38.732 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:50:38.732 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:50:38.732 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: legacy client fscid: 1 2024-02-10T06:50:38.732 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-02-10T06:50:38.732 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Filesystem 'foofs' (1) 2024-02-10T06:50:38.732 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: fs_name foofs 2024-02-10T06:50:38.732 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: epoch 5 2024-02-10T06:50:38.732 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-10T06:50:38.732 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: created 2024-02-10T06:44:02.111049+0000 2024-02-10T06:50:38.732 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: modified 2024-02-10T06:44:09.127989+0000 2024-02-10T06:50:38.733 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: tableserver 0 2024-02-10T06:50:38.733 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: root 0 2024-02-10T06:50:38.733 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: session_timeout 60 2024-02-10T06:50:38.733 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: session_autoclose 300 2024-02-10T06:50:38.733 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: max_file_size 1099511627776 2024-02-10T06:50:38.733 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: max_xattr_size 65536 2024-02-10T06:50:38.733 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: required_client_features {} 2024-02-10T06:50:38.733 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: last_failure 0 2024-02-10T06:50:38.733 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: last_failure_osd_epoch 0 2024-02-10T06:50:38.733 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:50:38.733 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: max_mds 1 2024-02-10T06:50:38.733 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: in 0 2024-02-10T06:50:38.733 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: up {0=14498} 2024-02-10T06:50:38.733 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: failed 2024-02-10T06:50:38.733 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: damaged 2024-02-10T06:50:38.734 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: stopped 2024-02-10T06:50:38.734 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: data_pools [3] 2024-02-10T06:50:38.734 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: metadata_pool 2 2024-02-10T06:50:38.734 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: inline_data disabled 2024-02-10T06:50:38.734 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: balancer 2024-02-10T06:50:38.734 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: bal_rank_mask -1 2024-02-10T06:50:38.734 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: standby_count_wanted 1 2024-02-10T06:50:38.734 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: [mds.foofs.smithi008.vfwcte{0:14498} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.8:6834/1608640408,v1:172.21.15.8:6835/1608640408] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:50:38.734 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-02-10T06:50:38.734 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-02-10T06:50:38.734 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: Standby daemons: 2024-02-10T06:50:38.734 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: 2024-02-10T06:50:38.734 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout: [mds.foofs.smithi029.bwmcyt{-1:24307} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.29:6832/1338152565,v1:172.21.15.29:6833/1338152565] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:50:38.734 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: mon.smithi029@-1(???).osd e73 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-10T06:50:38.734 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: mon.smithi029@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T06:50:38.735 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: mon.smithi029@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T06:50:38.735 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: mon.smithi029@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T06:50:38.735 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 ceph-mon[53210]: mon.smithi029@-1(???).paxosservice(auth 1..30) refresh upgraded, format 0 -> 3 2024-02-10T06:50:38.735 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 bash[53191]: 9492552149b2e4b58d8a07a94a69adfc7dce601716d19403d34b90b61a5a14ba 2024-02-10T06:50:38.735 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:38 smithi029 systemd[1]: Started Ceph mon.smithi029 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:50:39.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 181730 11.42 MB/sec execute 240 sec latency 27.735 ms 2024-02-10T06:50:39.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:39 smithi008 ceph-mon[66415]: pgmap v12: 97 pgs: 97 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.3 MiB/s wr, 547 op/s 2024-02-10T06:50:40.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:39 smithi029 ceph-mon[53210]: pgmap v12: 97 pgs: 97 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.3 MiB/s wr, 547 op/s 2024-02-10T06:50:40.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 182080 11.41 MB/sec execute 241 sec latency 34.583 ms 2024-02-10T06:50:41.099 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:40 smithi008 ceph-mon[66415]: overall HEALTH_OK 2024-02-10T06:50:41.099 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:40 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:50:41.099 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:40 smithi008 ceph-mon[66415]: mon.smithi029 calling monitor election 2024-02-10T06:50:41.099 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:40 smithi008 ceph-mon[66415]: mon.smithi008 calling monitor election 2024-02-10T06:50:41.099 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:40 smithi008 ceph-mon[66415]: mon.smithi008 is new leader, mons smithi008,smithi029 in quorum (ranks 0,1) 2024-02-10T06:50:41.099 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:40 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:50:41.100 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:40 smithi008 ceph-mon[66415]: monmap e3: 2 mons at {smithi008=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0],smithi029=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:50:41.100 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:40 smithi008 ceph-mon[66415]: fsmap foofs:1 {0=foofs.smithi008.vfwcte=up:active} 1 up:standby 2024-02-10T06:50:41.100 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:40 smithi008 ceph-mon[66415]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T06:50:41.100 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:40 smithi008 ceph-mon[66415]: mgrmap e40: smithi008.pxifbt(active, since 21s), standbys: smithi029.fuvrtf 2024-02-10T06:50:41.100 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:40 smithi008 ceph-mon[66415]: overall HEALTH_OK 2024-02-10T06:50:41.100 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:40 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:41.100 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:40 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:41.100 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:40 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:50:41.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:40 smithi029 ceph-mon[53210]: overall HEALTH_OK 2024-02-10T06:50:41.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:40 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-02-10T06:50:41.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:40 smithi029 ceph-mon[53210]: mon.smithi029 calling monitor election 2024-02-10T06:50:41.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:40 smithi029 ceph-mon[53210]: mon.smithi008 calling monitor election 2024-02-10T06:50:41.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:40 smithi029 ceph-mon[53210]: mon.smithi008 is new leader, mons smithi008,smithi029 in quorum (ranks 0,1) 2024-02-10T06:50:41.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:40 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-02-10T06:50:41.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:40 smithi029 ceph-mon[53210]: monmap e3: 2 mons at {smithi008=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0],smithi029=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:50:41.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:40 smithi029 ceph-mon[53210]: fsmap foofs:1 {0=foofs.smithi008.vfwcte=up:active} 1 up:standby 2024-02-10T06:50:41.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:40 smithi029 ceph-mon[53210]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T06:50:41.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:40 smithi029 ceph-mon[53210]: mgrmap e40: smithi008.pxifbt(active, since 21s), standbys: smithi029.fuvrtf 2024-02-10T06:50:41.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:40 smithi029 ceph-mon[53210]: overall HEALTH_OK 2024-02-10T06:50:41.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:40 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:41.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:40 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:41.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:40 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:50:41.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 182524 11.42 MB/sec execute 242 sec latency 46.986 ms 2024-02-10T06:50:42.053 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:41 smithi029 ceph-mon[53210]: pgmap v14: 97 pgs: 97 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.1 MiB/s wr, 694 op/s 2024-02-10T06:50:42.112 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:41 smithi008 ceph-mon[66415]: pgmap v14: 97 pgs: 97 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.1 MiB/s wr, 694 op/s 2024-02-10T06:50:42.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 182964 11.42 MB/sec execute 243 sec latency 34.537 ms 2024-02-10T06:50:43.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 183320 11.39 MB/sec execute 244 sec latency 34.053 ms 2024-02-10T06:50:44.128 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:43 smithi029 ceph-mon[53210]: pgmap v15: 97 pgs: 97 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.0 MiB/s wr, 503 op/s 2024-02-10T06:50:44.147 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:43 smithi008 ceph-mon[66415]: pgmap v15: 97 pgs: 97 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.0 MiB/s wr, 503 op/s 2024-02-10T06:50:44.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 183632 11.39 MB/sec execute 245 sec latency 68.606 ms 2024-02-10T06:50:45.155 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:45 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:45.155 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:45 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:45.155 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:45 smithi008 ceph-mon[66415]: pgmap v16: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 6.6 MiB/s wr, 597 op/s 2024-02-10T06:50:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:45 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:45 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:45 smithi029 ceph-mon[53210]: pgmap v16: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 6.6 MiB/s wr, 597 op/s 2024-02-10T06:50:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 184122 11.42 MB/sec execute 246 sec latency 34.366 ms 2024-02-10T06:50:46.428 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:46 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:46.428 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:46 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:46.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 184569 11.43 MB/sec execute 247 sec latency 57.918 ms 2024-02-10T06:50:46.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:46 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:46.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:46 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:47.439 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:47 smithi008 ceph-mon[66415]: pgmap v17: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 5.8 MiB/s wr, 551 op/s 2024-02-10T06:50:47.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 184969 11.42 MB/sec execute 248 sec latency 53.109 ms 2024-02-10T06:50:47.687 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:47 smithi029 ceph-mon[53210]: pgmap v17: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 5.8 MiB/s wr, 551 op/s 2024-02-10T06:50:48.474 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:50:48.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 185373 11.40 MB/sec execute 249 sec latency 45.742 ms 2024-02-10T06:50:48.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:50:49.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 185610 11.38 MB/sec execute 250 sec latency 81.039 ms 2024-02-10T06:50:49.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:49 smithi008 ceph-mon[66415]: pgmap v18: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.1 MiB/s wr, 506 op/s 2024-02-10T06:50:49.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:49.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:49.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:49.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:50:49.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:49.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:49.773 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:50:49.773 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:50:49.773 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:49.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:49 smithi029 ceph-mon[53210]: pgmap v18: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.1 MiB/s wr, 506 op/s 2024-02-10T06:50:49.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:49.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:49.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:49.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:50:49.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:49.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:49.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:50:49.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:50:49.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:50.642 INFO:teuthology.orchestra.run.smithi008.stdout: 5 185861 11.37 MB/sec execute 251 sec latency 94.311 ms 2024-02-10T06:50:50.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:50 smithi008 ceph-mon[66415]: Reconfiguring mon.smithi008 (monmap changed)... 2024-02-10T06:50:50.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:50 smithi008 ceph-mon[66415]: Reconfiguring daemon mon.smithi008 on smithi008 2024-02-10T06:50:50.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:50 smithi029 ceph-mon[53210]: Reconfiguring mon.smithi008 (monmap changed)... 2024-02-10T06:50:50.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:50 smithi029 ceph-mon[53210]: Reconfiguring daemon mon.smithi008 on smithi008 2024-02-10T06:50:51.644 INFO:teuthology.orchestra.run.smithi008.stdout: 5 186121 11.35 MB/sec execute 252 sec latency 85.029 ms 2024-02-10T06:50:51.650 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:51 smithi008 ceph-mon[66415]: pgmap v19: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 754 op/s 2024-02-10T06:50:51.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:51 smithi029 ceph-mon[53210]: pgmap v19: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 754 op/s 2024-02-10T06:50:52.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 186473 11.34 MB/sec execute 253 sec latency 50.595 ms 2024-02-10T06:50:52.886 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:50:53.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 186823 11.32 MB/sec execute 254 sec latency 70.465 ms 2024-02-10T06:50:53.957 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:53 smithi008 ceph-mon[66415]: pgmap v20: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.0 MiB/s wr, 494 op/s 2024-02-10T06:50:53.958 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:53 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:53.958 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:53 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:53.958 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:53 smithi008 ceph-mon[66415]: Reconfiguring mgr.smithi008.pxifbt (monmap changed)... 2024-02-10T06:50:53.958 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:53 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.pxifbt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:50:53.958 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:53 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:50:53.958 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:53 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:53.958 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:53 smithi008 ceph-mon[66415]: Reconfiguring daemon mgr.smithi008.pxifbt on smithi008 2024-02-10T06:50:53.958 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:53 smithi008 ceph-mon[66415]: from='client.44101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:50:53.959 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:50:53.959 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (3m) 26s ago 12m 16.7M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:50:53.959 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (12m) 26s ago 12m 6405k - 17.2.0 e1d6a67b021e 5fa2e5bdd7cc 2024-02-10T06:50:53.960 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (12m) 9s ago 12m 6405k - 17.2.0 e1d6a67b021e 7527ea77b5cc 2024-02-10T06:50:53.960 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (2m) 26s ago 12m 42.6M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:50:53.960 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (6m) 26s ago 6m 37.9M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:50:53.960 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (6m) 9s ago 6m 14.2M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:50:53.960 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (2m) 26s ago 14m 478M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:50:53.960 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (65s) 9s ago 12m 447M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:50:53.961 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (46s) 26s ago 14m 33.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:50:53.961 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (16s) 9s ago 12m 24.7M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:50:53.961 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (6m) 26s ago 6m 274M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:50:53.961 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (6m) 9s ago 6m 84.3M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:50:53.961 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (2m) 26s ago 12m 9265k - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:50:53.961 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (2m) 9s ago 11m 21.4M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:50:53.961 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (11m) 26s ago 11m 337M 1937M 17.2.0 e1d6a67b021e 139b7a92bfc4 2024-02-10T06:50:53.961 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (10m) 26s ago 10m 523M 1937M 17.2.0 e1d6a67b021e 0f1119e17a52 2024-02-10T06:50:53.962 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (10m) 26s ago 10m 403M 1937M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:50:53.962 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (10m) 26s ago 10m 523M 1937M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:50:53.962 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (9m) 9s ago 9m 551M 2705M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:50:53.962 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (9m) 9s ago 9m 402M 2705M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:50:53.962 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (8m) 9s ago 8m 577M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:50:53.962 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (8m) 9s ago 8m 484M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:50:53.962 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (72s) 26s ago 11m 55.8M - 2.33.4 514e6a882f6e 193819b86dea 2024-02-10T06:50:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:53 smithi029 ceph-mon[53210]: pgmap v20: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.0 MiB/s wr, 494 op/s 2024-02-10T06:50:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:53 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:53 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:53 smithi029 ceph-mon[53210]: Reconfiguring mgr.smithi008.pxifbt (monmap changed)... 2024-02-10T06:50:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:53 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.pxifbt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:50:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:53 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:50:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:53 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:53 smithi029 ceph-mon[53210]: Reconfiguring daemon mgr.smithi008.pxifbt on smithi008 2024-02-10T06:50:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:53 smithi029 ceph-mon[53210]: from='client.44101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:50:54.546 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:50:54.547 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:50:54.547 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:50:54.547 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:50:54.547 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:50:54.547 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:50:54.547 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:50:54.547 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:50:54.547 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:50:54.547 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:50:54.547 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:50:54.547 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:50:54.547 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:50:54.548 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:50:54.548 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-10T06:50:54.548 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 4 2024-02-10T06:50:54.548 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:50:54.548 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:50:54.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 187304 11.34 MB/sec execute 255 sec latency 51.032 ms 2024-02-10T06:50:54.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:54 smithi029 ceph-mon[53210]: from='client.44103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:50:54.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:54 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/3564473542' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:50:54.979 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:54 smithi008 ceph-mon[66415]: from='client.44103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:50:54.979 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:54 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/3564473542' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:50:55.042 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:50:55.042 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:50:55.043 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:50:55.043 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:50:55.043 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:50:55.043 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:50:55.043 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-02-10T06:50:55.043 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:50:55.043 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "4/23 daemons upgraded", 2024-02-10T06:50:55.043 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading mon daemons", 2024-02-10T06:50:55.043 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:50:55.043 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:50:55.619 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:50:55.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 187726 11.35 MB/sec execute 256 sec latency 47.292 ms 2024-02-10T06:50:55.984 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:55 smithi008 ceph-mon[66415]: from='client.44105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:50:55.984 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:55 smithi008 ceph-mon[66415]: pgmap v21: 97 pgs: 97 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 6.7 MiB/s wr, 675 op/s 2024-02-10T06:50:55.984 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:55 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/3527482548' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:50:55.984 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:55 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:55.984 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:55 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:55.984 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:55 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:50:55.985 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:55 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:55 smithi029 ceph-mon[53210]: from='client.44105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:50:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:55 smithi029 ceph-mon[53210]: pgmap v21: 97 pgs: 97 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 6.7 MiB/s wr, 675 op/s 2024-02-10T06:50:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:55 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/3527482548' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:50:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:55 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:55 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:55 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:50:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:55 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:56.642 INFO:teuthology.orchestra.run.smithi008.stdout: 5 188024 11.34 MB/sec execute 257 sec latency 43.302 ms 2024-02-10T06:50:56.997 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:56 smithi008 ceph-mon[66415]: from='client.34170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:50:56.997 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:56 smithi008 ceph-mon[66415]: Reconfiguring crash.smithi008 (monmap changed)... 2024-02-10T06:50:56.997 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:56 smithi008 ceph-mon[66415]: Reconfiguring daemon crash.smithi008 on smithi008 2024-02-10T06:50:57.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:56 smithi029 ceph-mon[53210]: from='client.34170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:50:57.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:56 smithi029 ceph-mon[53210]: Reconfiguring crash.smithi008 (monmap changed)... 2024-02-10T06:50:57.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:56 smithi029 ceph-mon[53210]: Reconfiguring daemon crash.smithi008 on smithi008 2024-02-10T06:50:57.641 INFO:teuthology.orchestra.run.smithi008.stdout: 5 188280 11.33 MB/sec execute 258 sec latency 56.164 ms 2024-02-10T06:50:58.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:57 smithi008 ceph-mon[66415]: pgmap v22: 97 pgs: 97 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 5.4 MiB/s wr, 594 op/s 2024-02-10T06:50:58.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:57 smithi029 ceph-mon[53210]: pgmap v22: 97 pgs: 97 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 5.4 MiB/s wr, 594 op/s 2024-02-10T06:50:58.642 INFO:teuthology.orchestra.run.smithi008.stdout: 5 188588 11.31 MB/sec execute 259 sec latency 52.167 ms 2024-02-10T06:50:59.303 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:59 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:59.303 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:59 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:59.303 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:59 smithi008 ceph-mon[66415]: Reconfiguring osd.0 (monmap changed)... 2024-02-10T06:50:59.303 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:59 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T06:50:59.303 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:59 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:59.303 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:59 smithi008 ceph-mon[66415]: Reconfiguring daemon osd.0 on smithi008 2024-02-10T06:50:59.303 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:50:59 smithi008 ceph-mon[66415]: pgmap v23: 97 pgs: 97 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 441 op/s 2024-02-10T06:50:59.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:59 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:59.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:59 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:50:59.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:59 smithi029 ceph-mon[53210]: Reconfiguring osd.0 (monmap changed)... 2024-02-10T06:50:59.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:59 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T06:50:59.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:59 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:50:59.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:59 smithi029 ceph-mon[53210]: Reconfiguring daemon osd.0 on smithi008 2024-02-10T06:50:59.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:50:59 smithi029 ceph-mon[53210]: pgmap v23: 97 pgs: 97 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 441 op/s 2024-02-10T06:50:59.642 INFO:teuthology.orchestra.run.smithi008.stdout: 5 188873 11.28 MB/sec execute 260 sec latency 43.105 ms 2024-02-10T06:51:00.643 INFO:teuthology.orchestra.run.smithi008.stdout: 5 189184 11.27 MB/sec execute 261 sec latency 68.070 ms 2024-02-10T06:51:01.643 INFO:teuthology.orchestra.run.smithi008.stdout: 5 189593 11.28 MB/sec execute 262 sec latency 41.927 ms 2024-02-10T06:51:01.897 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:01 smithi008 ceph-mon[66415]: pgmap v24: 97 pgs: 97 active+clean; 347 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.5 MiB/s wr, 641 op/s 2024-02-10T06:51:01.897 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:01 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:01.897 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:01 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:01.897 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:01 smithi008 ceph-mon[66415]: Reconfiguring osd.1 (monmap changed)... 2024-02-10T06:51:01.897 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:01 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T06:51:01.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:01 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:01.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:01 smithi008 ceph-mon[66415]: Reconfiguring daemon osd.1 on smithi008 2024-02-10T06:51:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:01 smithi029 ceph-mon[53210]: pgmap v24: 97 pgs: 97 active+clean; 347 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.5 MiB/s wr, 641 op/s 2024-02-10T06:51:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:01 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:01 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:01 smithi029 ceph-mon[53210]: Reconfiguring osd.1 (monmap changed)... 2024-02-10T06:51:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:01 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T06:51:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:01 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:01.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:01 smithi029 ceph-mon[53210]: Reconfiguring daemon osd.1 on smithi008 2024-02-10T06:51:02.643 INFO:teuthology.orchestra.run.smithi008.stdout: 5 190093 11.28 MB/sec execute 263 sec latency 71.884 ms 2024-02-10T06:51:03.647 INFO:teuthology.orchestra.run.smithi008.stdout: 5 190639 11.29 MB/sec execute 264 sec latency 47.480 ms 2024-02-10T06:51:03.893 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:03 smithi008 ceph-mon[66415]: pgmap v25: 97 pgs: 97 active+clean; 347 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.9 MiB/s wr, 393 op/s 2024-02-10T06:51:03.893 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:51:03.894 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:03.894 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:03.894 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T06:51:03.894 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:03 smithi029 ceph-mon[53210]: pgmap v25: 97 pgs: 97 active+clean; 347 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.9 MiB/s wr, 393 op/s 2024-02-10T06:51:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:51:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T06:51:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:04.650 INFO:teuthology.orchestra.run.smithi008.stdout: 5 191219 11.32 MB/sec execute 265 sec latency 39.417 ms 2024-02-10T06:51:04.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:04 smithi008 ceph-mon[66415]: Reconfiguring osd.2 (monmap changed)... 2024-02-10T06:51:04.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:04 smithi008 ceph-mon[66415]: Reconfiguring daemon osd.2 on smithi008 2024-02-10T06:51:04.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:04 smithi029 ceph-mon[53210]: Reconfiguring osd.2 (monmap changed)... 2024-02-10T06:51:04.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:04 smithi029 ceph-mon[53210]: Reconfiguring daemon osd.2 on smithi008 2024-02-10T06:51:05.650 INFO:teuthology.orchestra.run.smithi008.stdout: 5 191669 11.32 MB/sec execute 266 sec latency 31.904 ms 2024-02-10T06:51:05.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:05 smithi008 ceph-mon[66415]: pgmap v26: 97 pgs: 97 active+clean; 351 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 6.4 MiB/s wr, 644 op/s 2024-02-10T06:51:05.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:05 smithi029 ceph-mon[53210]: pgmap v26: 97 pgs: 97 active+clean; 351 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 6.4 MiB/s wr, 644 op/s 2024-02-10T06:51:06.650 INFO:teuthology.orchestra.run.smithi008.stdout: 5 192120 11.32 MB/sec execute 267 sec latency 72.880 ms 2024-02-10T06:51:07.427 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:07 smithi008 ceph-mon[66415]: pgmap v27: 97 pgs: 97 active+clean; 351 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.0 MiB/s wr, 484 op/s 2024-02-10T06:51:07.427 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:07 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:07.427 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:07 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:07.427 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:07 smithi008 ceph-mon[66415]: Reconfiguring osd.3 (monmap changed)... 2024-02-10T06:51:07.427 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:07 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T06:51:07.427 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:07 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:07.427 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:07 smithi008 ceph-mon[66415]: Reconfiguring daemon osd.3 on smithi008 2024-02-10T06:51:07.650 INFO:teuthology.orchestra.run.smithi008.stdout: 5 192373 11.30 MB/sec execute 268 sec latency 101.061 ms 2024-02-10T06:51:07.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:07 smithi029 ceph-mon[53210]: pgmap v27: 97 pgs: 97 active+clean; 351 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.0 MiB/s wr, 484 op/s 2024-02-10T06:51:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:07 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:07 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:07 smithi029 ceph-mon[53210]: Reconfiguring osd.3 (monmap changed)... 2024-02-10T06:51:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:07 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T06:51:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:07 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:07 smithi029 ceph-mon[53210]: Reconfiguring daemon osd.3 on smithi008 2024-02-10T06:51:08.650 INFO:teuthology.orchestra.run.smithi008.stdout: 5 192612 11.29 MB/sec execute 269 sec latency 55.001 ms 2024-02-10T06:51:09.650 INFO:teuthology.orchestra.run.smithi008.stdout: 5 192909 11.29 MB/sec execute 270 sec latency 61.617 ms 2024-02-10T06:51:09.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:09 smithi008 ceph-mon[66415]: pgmap v28: 97 pgs: 97 active+clean; 351 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.8 MiB/s wr, 471 op/s 2024-02-10T06:51:09.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:09 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:09.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:09 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:09.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:09 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.vfwcte", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:51:09.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:09 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:09.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:09 smithi029 ceph-mon[53210]: pgmap v28: 97 pgs: 97 active+clean; 351 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.8 MiB/s wr, 471 op/s 2024-02-10T06:51:09.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:09 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:09.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:09 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:09.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:09 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.vfwcte", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:51:09.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:09 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:10.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 193165 11.26 MB/sec execute 271 sec latency 71.307 ms 2024-02-10T06:51:10.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:10 smithi008 ceph-mon[66415]: Reconfiguring mds.foofs.smithi008.vfwcte (monmap changed)... 2024-02-10T06:51:10.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:10 smithi008 ceph-mon[66415]: Reconfiguring daemon mds.foofs.smithi008.vfwcte on smithi008 2024-02-10T06:51:10.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:10 smithi029 ceph-mon[53210]: Reconfiguring mds.foofs.smithi008.vfwcte (monmap changed)... 2024-02-10T06:51:10.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:10 smithi029 ceph-mon[53210]: Reconfiguring daemon mds.foofs.smithi008.vfwcte on smithi008 2024-02-10T06:51:11.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 193495 11.24 MB/sec execute 272 sec latency 123.837 ms 2024-02-10T06:51:11.750 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:11 smithi008 ceph-mon[66415]: pgmap v29: 97 pgs: 97 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 701 op/s 2024-02-10T06:51:11.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:11 smithi029 ceph-mon[53210]: pgmap v29: 97 pgs: 97 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 701 op/s 2024-02-10T06:51:12.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 193749 11.22 MB/sec execute 273 sec latency 53.021 ms 2024-02-10T06:51:13.029 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:12 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:13.029 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:12 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:13.029 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:12 smithi008 ceph-mon[66415]: Reconfiguring crash.smithi029 (monmap changed)... 2024-02-10T06:51:13.029 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:12 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:51:13.029 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:12 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:13.029 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:12 smithi008 ceph-mon[66415]: Reconfiguring daemon crash.smithi029 on smithi029 2024-02-10T06:51:13.029 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:12 smithi008 ceph-mon[66415]: pgmap v30: 97 pgs: 97 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.0 MiB/s wr, 502 op/s 2024-02-10T06:51:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:12 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:12 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:12 smithi029 ceph-mon[53210]: Reconfiguring crash.smithi029 (monmap changed)... 2024-02-10T06:51:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:12 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:51:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:12 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:12 smithi029 ceph-mon[53210]: Reconfiguring daemon crash.smithi029 on smithi029 2024-02-10T06:51:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:12 smithi029 ceph-mon[53210]: pgmap v30: 97 pgs: 97 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.0 MiB/s wr, 502 op/s 2024-02-10T06:51:13.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 194065 11.21 MB/sec execute 274 sec latency 60.068 ms 2024-02-10T06:51:14.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 194333 11.20 MB/sec execute 275 sec latency 48.248 ms 2024-02-10T06:51:14.822 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:14.822 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:14.822 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:14 smithi008 ceph-mon[66415]: Reconfiguring mgr.smithi029.fuvrtf (monmap changed)... 2024-02-10T06:51:14.822 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi029.fuvrtf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:51:14.822 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:51:14.822 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:14.822 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:14 smithi008 ceph-mon[66415]: Reconfiguring daemon mgr.smithi029.fuvrtf on smithi029 2024-02-10T06:51:14.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:14.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:14.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:14 smithi029 ceph-mon[53210]: Reconfiguring mgr.smithi029.fuvrtf (monmap changed)... 2024-02-10T06:51:14.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi029.fuvrtf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:51:14.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:51:14.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:14.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:14 smithi029 ceph-mon[53210]: Reconfiguring daemon mgr.smithi029.fuvrtf on smithi029 2024-02-10T06:51:15.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 194834 11.22 MB/sec execute 276 sec latency 61.565 ms 2024-02-10T06:51:15.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:15 smithi008 ceph-mon[66415]: pgmap v31: 97 pgs: 97 active+clean; 344 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.3 MiB/s wr, 589 op/s 2024-02-10T06:51:15.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:15 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:15.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:15 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:15.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:15 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:51:15.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:15 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:51:15.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:15 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:15.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:15 smithi029 ceph-mon[53210]: pgmap v31: 97 pgs: 97 active+clean; 344 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.3 MiB/s wr, 589 op/s 2024-02-10T06:51:15.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:15 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:15.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:15 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:15.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:15 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:51:15.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:15 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:51:15.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:15 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:16.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 195247 11.22 MB/sec execute 277 sec latency 32.084 ms 2024-02-10T06:51:16.837 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:16 smithi008 ceph-mon[66415]: Reconfiguring mon.smithi029 (monmap changed)... 2024-02-10T06:51:16.837 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:16 smithi008 ceph-mon[66415]: Reconfiguring daemon mon.smithi029 on smithi029 2024-02-10T06:51:16.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:16 smithi029 ceph-mon[53210]: Reconfiguring mon.smithi029 (monmap changed)... 2024-02-10T06:51:16.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:16 smithi029 ceph-mon[53210]: Reconfiguring daemon mon.smithi029 on smithi029 2024-02-10T06:51:17.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 195602 11.22 MB/sec execute 278 sec latency 100.988 ms 2024-02-10T06:51:17.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:17 smithi008 ceph-mon[66415]: pgmap v32: 97 pgs: 97 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 4.4 MiB/s wr, 428 op/s 2024-02-10T06:51:17.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:17.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:17.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T06:51:17.867 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:17 smithi029 ceph-mon[53210]: pgmap v32: 97 pgs: 97 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 4.4 MiB/s wr, 428 op/s 2024-02-10T06:51:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T06:51:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:18.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 195881 11.19 MB/sec execute 279 sec latency 42.244 ms 2024-02-10T06:51:18.880 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:18 smithi008 ceph-mon[66415]: Reconfiguring osd.4 (monmap changed)... 2024-02-10T06:51:18.880 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:18 smithi008 ceph-mon[66415]: Reconfiguring daemon osd.4 on smithi029 2024-02-10T06:51:18.880 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:51:18.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:18 smithi029 ceph-mon[53210]: Reconfiguring osd.4 (monmap changed)... 2024-02-10T06:51:18.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:18 smithi029 ceph-mon[53210]: Reconfiguring daemon osd.4 on smithi029 2024-02-10T06:51:18.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:51:19.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 196174 11.18 MB/sec execute 280 sec latency 44.112 ms 2024-02-10T06:51:19.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:19 smithi008 ceph-mon[66415]: pgmap v33: 97 pgs: 97 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.0 MiB/s wr, 407 op/s 2024-02-10T06:51:19.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:19 smithi029 ceph-mon[53210]: pgmap v33: 97 pgs: 97 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.0 MiB/s wr, 407 op/s 2024-02-10T06:51:20.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 196463 11.19 MB/sec execute 281 sec latency 63.918 ms 2024-02-10T06:51:20.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:20 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:20.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:20 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:20.931 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:20 smithi008 ceph-mon[66415]: Reconfiguring osd.5 (monmap changed)... 2024-02-10T06:51:20.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:20 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T06:51:20.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:20 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:20.932 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:20 smithi008 ceph-mon[66415]: Reconfiguring daemon osd.5 on smithi029 2024-02-10T06:51:21.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:20 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:21.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:20 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:21.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:20 smithi029 ceph-mon[53210]: Reconfiguring osd.5 (monmap changed)... 2024-02-10T06:51:21.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:20 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T06:51:21.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:20 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:21.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:20 smithi029 ceph-mon[53210]: Reconfiguring daemon osd.5 on smithi029 2024-02-10T06:51:21.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 196966 11.18 MB/sec execute 282 sec latency 76.122 ms 2024-02-10T06:51:21.947 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:21 smithi008 ceph-mon[66415]: pgmap v34: 97 pgs: 97 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.3 MiB/s wr, 630 op/s 2024-02-10T06:51:21.948 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:21 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:21.948 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:21 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:21.948 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:21 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T06:51:21.948 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:21 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:21.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:21 smithi029 ceph-mon[53210]: pgmap v34: 97 pgs: 97 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.3 MiB/s wr, 630 op/s 2024-02-10T06:51:21.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:21 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:21.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:21 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:21.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:21 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T06:51:21.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:21 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:22.655 INFO:teuthology.orchestra.run.smithi008.stdout: 5 197357 11.17 MB/sec execute 283 sec latency 55.230 ms 2024-02-10T06:51:22.970 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:22 smithi008 ceph-mon[66415]: Reconfiguring osd.6 (monmap changed)... 2024-02-10T06:51:22.970 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:22 smithi008 ceph-mon[66415]: Reconfiguring daemon osd.6 on smithi029 2024-02-10T06:51:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:22 smithi029 ceph-mon[53210]: Reconfiguring osd.6 (monmap changed)... 2024-02-10T06:51:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:22 smithi029 ceph-mon[53210]: Reconfiguring daemon osd.6 on smithi029 2024-02-10T06:51:23.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 197705 11.17 MB/sec execute 284 sec latency 106.127 ms 2024-02-10T06:51:24.011 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:23 smithi008 ceph-mon[66415]: pgmap v35: 97 pgs: 97 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 400 op/s 2024-02-10T06:51:24.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:23 smithi029 ceph-mon[53210]: pgmap v35: 97 pgs: 97 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 400 op/s 2024-02-10T06:51:24.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 198048 11.16 MB/sec execute 285 sec latency 99.000 ms 2024-02-10T06:51:25.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 198484 11.18 MB/sec execute 286 sec latency 37.526 ms 2024-02-10T06:51:25.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:25 smithi008 ceph-mon[66415]: pgmap v36: 97 pgs: 97 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 6.3 MiB/s wr, 622 op/s 2024-02-10T06:51:25.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:25 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:25.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:25 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:25.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:25 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T06:51:25.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:25 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:26.060 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:25 smithi029 ceph-mon[53210]: pgmap v36: 97 pgs: 97 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 6.3 MiB/s wr, 622 op/s 2024-02-10T06:51:26.060 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:25 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:26.061 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:25 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:26.061 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:25 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T06:51:26.061 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:25 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:26.227 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:51:26.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 198834 11.17 MB/sec execute 287 sec latency 44.987 ms 2024-02-10T06:51:26.819 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:26 smithi008 ceph-mon[66415]: Reconfiguring osd.7 (monmap changed)... 2024-02-10T06:51:26.819 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:26 smithi008 ceph-mon[66415]: Reconfiguring daemon osd.7 on smithi029 2024-02-10T06:51:27.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:26 smithi029 ceph-mon[53210]: Reconfiguring osd.7 (monmap changed)... 2024-02-10T06:51:27.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:26 smithi029 ceph-mon[53210]: Reconfiguring daemon osd.7 on smithi029 2024-02-10T06:51:27.341 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:51:27.341 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (3m) 59s ago 13m 16.7M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:51:27.341 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (13m) 59s ago 13m 6405k - 17.2.0 e1d6a67b021e 5fa2e5bdd7cc 2024-02-10T06:51:27.341 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (12m) 43s ago 12m 6405k - 17.2.0 e1d6a67b021e 7527ea77b5cc 2024-02-10T06:51:27.342 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (3m) 59s ago 13m 42.6M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:51:27.342 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (7m) 59s ago 7m 37.9M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:51:27.342 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (7m) 43s ago 7m 14.2M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:51:27.342 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (2m) 59s ago 14m 478M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:51:27.342 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (98s) 43s ago 12m 447M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:51:27.342 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (79s) 59s ago 14m 33.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:51:27.342 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (49s) 43s ago 12m 24.7M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:51:27.342 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (7m) 59s ago 7m 274M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:51:27.342 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (6m) 43s ago 6m 84.3M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:51:27.342 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (3m) 59s ago 13m 9265k - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:51:27.342 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (2m) 43s ago 12m 21.4M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:51:27.342 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (12m) 59s ago 12m 337M 1937M 17.2.0 e1d6a67b021e 139b7a92bfc4 2024-02-10T06:51:27.342 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (11m) 59s ago 11m 523M 1937M 17.2.0 e1d6a67b021e 0f1119e17a52 2024-02-10T06:51:27.342 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (11m) 59s ago 11m 403M 1937M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:51:27.342 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (10m) 59s ago 10m 523M 1937M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:51:27.343 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (10m) 43s ago 10m 551M 2705M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:51:27.343 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (9m) 43s ago 9m 402M 2705M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:51:27.343 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (9m) 43s ago 9m 577M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:51:27.343 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (8m) 43s ago 8m 484M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:51:27.343 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (105s) 59s ago 12m 55.8M - 2.33.4 514e6a882f6e 193819b86dea 2024-02-10T06:51:27.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 199276 11.16 MB/sec execute 288 sec latency 49.757 ms 2024-02-10T06:51:28.022 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:51:28.022 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:51:28.022 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:51:28.022 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:51:28.022 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:51:28.023 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:51:28.023 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:51:28.023 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:51:28.023 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:51:28.023 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:51:28.023 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:51:28.023 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:51:28.023 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:51:28.023 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:51:28.023 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-10T06:51:28.023 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 4 2024-02-10T06:51:28.024 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:51:28.024 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:51:28.086 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:27 smithi008 ceph-mon[66415]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:51:28.086 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:27 smithi008 ceph-mon[66415]: pgmap v37: 97 pgs: 97 active+clean; 345 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.1 MiB/s wr, 536 op/s 2024-02-10T06:51:28.086 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:27 smithi008 ceph-mon[66415]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:51:28.086 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:28.086 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:28.086 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi029.bwmcyt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:51:28.087 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:27 smithi029 ceph-mon[53210]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:51:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:27 smithi029 ceph-mon[53210]: pgmap v37: 97 pgs: 97 active+clean; 345 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.1 MiB/s wr, 536 op/s 2024-02-10T06:51:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:27 smithi029 ceph-mon[53210]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:51:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi029.bwmcyt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:51:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:28.544 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:51:28.545 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:51:28.545 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:51:28.545 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:51:28.545 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:51:28.545 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:51:28.545 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-02-10T06:51:28.545 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:51:28.545 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "4/23 daemons upgraded", 2024-02-10T06:51:28.545 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading mon daemons", 2024-02-10T06:51:28.545 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:51:28.545 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:51:28.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 199581 11.15 MB/sec execute 289 sec latency 34.669 ms 2024-02-10T06:51:29.048 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:28 smithi008 ceph-mon[66415]: Reconfiguring mds.foofs.smithi029.bwmcyt (monmap changed)... 2024-02-10T06:51:29.049 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:28 smithi008 ceph-mon[66415]: Reconfiguring daemon mds.foofs.smithi029.bwmcyt on smithi029 2024-02-10T06:51:29.049 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:28 smithi008 ceph-mon[66415]: from='client.34186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:51:29.049 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:28 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/2522810791' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:51:29.119 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:51:29.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:28 smithi029 ceph-mon[53210]: Reconfiguring mds.foofs.smithi029.bwmcyt (monmap changed)... 2024-02-10T06:51:29.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:28 smithi029 ceph-mon[53210]: Reconfiguring daemon mds.foofs.smithi029.bwmcyt on smithi029 2024-02-10T06:51:29.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:28 smithi029 ceph-mon[53210]: from='client.34186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:51:29.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:28 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/2522810791' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:51:29.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 199948 11.15 MB/sec execute 290 sec latency 176.062 ms 2024-02-10T06:51:29.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:29 smithi008 ceph-mon[66415]: pgmap v38: 97 pgs: 97 active+clean; 345 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.5 MiB/s wr, 446 op/s 2024-02-10T06:51:29.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:29 smithi008 ceph-mon[66415]: from='client.44123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:51:29.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:29 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/1748760170' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:51:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:29 smithi029 ceph-mon[53210]: pgmap v38: 97 pgs: 97 active+clean; 345 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.5 MiB/s wr, 446 op/s 2024-02-10T06:51:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:29 smithi029 ceph-mon[53210]: from='client.44123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:51:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:29 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/1748760170' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:51:30.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 200256 11.14 MB/sec execute 291 sec latency 79.880 ms 2024-02-10T06:51:30.987 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:30 smithi008 ceph-mon[66415]: pgmap v39: 97 pgs: 97 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.9 MiB/s wr, 678 op/s 2024-02-10T06:51:31.065 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:30 smithi029 ceph-mon[53210]: pgmap v39: 97 pgs: 97 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.9 MiB/s wr, 678 op/s 2024-02-10T06:51:31.652 INFO:teuthology.orchestra.run.smithi008.stdout: 5 200659 11.13 MB/sec execute 292 sec latency 46.887 ms 2024-02-10T06:51:32.264 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:32.264 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:32.264 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:51:32.264 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:51:32.265 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:51:32.265 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:32 smithi008 ceph-mon[66415]: Upgrade: Setting container_image for all mon 2024-02-10T06:51:32.265 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:32.265 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi008"}]: dispatch 2024-02-10T06:51:32.265 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi008"}]': finished 2024-02-10T06:51:32.265 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi029"}]: dispatch 2024-02-10T06:51:32.265 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi029"}]': finished 2024-02-10T06:51:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:51:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:51:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:51:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:32 smithi029 ceph-mon[53210]: Upgrade: Setting container_image for all mon 2024-02-10T06:51:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi008"}]: dispatch 2024-02-10T06:51:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi008"}]': finished 2024-02-10T06:51:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi029"}]: dispatch 2024-02-10T06:51:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi029"}]': finished 2024-02-10T06:51:32.653 INFO:teuthology.orchestra.run.smithi008.stdout: 5 200950 11.12 MB/sec execute 293 sec latency 40.691 ms 2024-02-10T06:51:33.276 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:33 smithi008 ceph-mon[66415]: pgmap v40: 97 pgs: 97 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.6 MiB/s wr, 455 op/s 2024-02-10T06:51:33.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:33 smithi029 ceph-mon[53210]: pgmap v40: 97 pgs: 97 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.6 MiB/s wr, 455 op/s 2024-02-10T06:51:33.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 201296 11.12 MB/sec execute 294 sec latency 71.684 ms 2024-02-10T06:51:34.290 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:34 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:51:34.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:34 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:51:34.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 201578 11.10 MB/sec execute 295 sec latency 60.724 ms 2024-02-10T06:51:35.316 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:35 smithi008 ceph-mon[66415]: Upgrade: Updating crash.smithi008 (1/2) 2024-02-10T06:51:35.317 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:35 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:35.317 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:35 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:51:35.317 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:35 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:35.317 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:35 smithi008 ceph-mon[66415]: Deploying daemon crash.smithi008 on smithi008 2024-02-10T06:51:35.317 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:35 smithi008 ceph-mon[66415]: pgmap v41: 97 pgs: 97 active+clean; 346 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 6.7 MiB/s wr, 663 op/s 2024-02-10T06:51:35.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:35 smithi029 ceph-mon[53210]: Upgrade: Updating crash.smithi008 (1/2) 2024-02-10T06:51:35.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:35 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:35.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:35 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:51:35.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:35 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:35.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:35 smithi029 ceph-mon[53210]: Deploying daemon crash.smithi008 on smithi008 2024-02-10T06:51:35.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:35 smithi029 ceph-mon[53210]: pgmap v41: 97 pgs: 97 active+clean; 346 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 6.7 MiB/s wr, 663 op/s 2024-02-10T06:51:35.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 201917 11.11 MB/sec execute 296 sec latency 62.952 ms 2024-02-10T06:51:36.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 202173 11.09 MB/sec execute 297 sec latency 45.889 ms 2024-02-10T06:51:37.448 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:37 smithi008 ceph-mon[66415]: pgmap v42: 97 pgs: 97 active+clean; 346 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.8 KiB/s rd, 4.5 MiB/s wr, 442 op/s 2024-02-10T06:51:37.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 202394 11.08 MB/sec execute 298 sec latency 80.381 ms 2024-02-10T06:51:37.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:37 smithi029 ceph-mon[53210]: pgmap v42: 97 pgs: 97 active+clean; 346 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.8 KiB/s rd, 4.5 MiB/s wr, 442 op/s 2024-02-10T06:51:38.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 202671 11.06 MB/sec execute 299 sec latency 77.026 ms 2024-02-10T06:51:39.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 202934 11.04 MB/sec execute 300 sec latency 50.775 ms 2024-02-10T06:51:39.749 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:39 smithi008 ceph-mon[66415]: pgmap v43: 97 pgs: 97 active+clean; 346 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.5 MiB/s wr, 441 op/s 2024-02-10T06:51:39.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:39 smithi029 ceph-mon[53210]: pgmap v43: 97 pgs: 97 active+clean; 346 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.5 MiB/s wr, 441 op/s 2024-02-10T06:51:40.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 203133 11.02 MB/sec execute 301 sec latency 47.227 ms 2024-02-10T06:51:41.115 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:41 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:41.115 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:41 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:41.115 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:41 smithi029 ceph-mon[53210]: pgmap v44: 97 pgs: 97 active+clean; 344 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.0 MiB/s wr, 603 op/s 2024-02-10T06:51:41.294 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:41 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:41.294 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:41 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:41.295 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:41 smithi008 ceph-mon[66415]: pgmap v44: 97 pgs: 97 active+clean; 344 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.0 MiB/s wr, 603 op/s 2024-02-10T06:51:41.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 203573 11.04 MB/sec execute 302 sec latency 48.648 ms 2024-02-10T06:51:42.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 204073 11.04 MB/sec execute 303 sec latency 40.124 ms 2024-02-10T06:51:43.257 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:43 smithi008 ceph-mon[66415]: Upgrade: Updating crash.smithi029 (2/2) 2024-02-10T06:51:43.258 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:43 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:43.258 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:43 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:51:43.258 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:43 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:43.258 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:43 smithi008 ceph-mon[66415]: Deploying daemon crash.smithi029 on smithi029 2024-02-10T06:51:43.258 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:43 smithi008 ceph-mon[66415]: pgmap v45: 97 pgs: 97 active+clean; 344 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.6 MiB/s wr, 371 op/s 2024-02-10T06:51:43.362 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:43 smithi029 ceph-mon[53210]: Upgrade: Updating crash.smithi029 (2/2) 2024-02-10T06:51:43.362 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:43 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:43.362 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:43 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:51:43.362 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:43 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:51:43.362 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:43 smithi029 ceph-mon[53210]: Deploying daemon crash.smithi029 on smithi029 2024-02-10T06:51:43.362 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:43 smithi029 ceph-mon[53210]: pgmap v45: 97 pgs: 97 active+clean; 344 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.6 MiB/s wr, 371 op/s 2024-02-10T06:51:43.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 204545 11.04 MB/sec execute 304 sec latency 40.879 ms 2024-02-10T06:51:44.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 205043 11.04 MB/sec execute 305 sec latency 31.315 ms 2024-02-10T06:51:45.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 205557 11.07 MB/sec execute 306 sec latency 66.054 ms 2024-02-10T06:51:45.818 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:45 smithi029 ceph-mon[53210]: pgmap v46: 97 pgs: 97 active+clean; 348 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 6.1 MiB/s wr, 617 op/s 2024-02-10T06:51:45.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:45 smithi008 ceph-mon[66415]: pgmap v46: 97 pgs: 97 active+clean; 348 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 6.1 MiB/s wr, 617 op/s 2024-02-10T06:51:46.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 205988 11.06 MB/sec execute 307 sec latency 42.671 ms 2024-02-10T06:51:47.630 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:47 smithi008 ceph-mon[66415]: pgmap v47: 97 pgs: 97 active+clean; 348 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 412 op/s 2024-02-10T06:51:47.630 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:47.630 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:47.630 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:51:47.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 206348 11.05 MB/sec execute 308 sec latency 43.236 ms 2024-02-10T06:51:47.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:47 smithi029 ceph-mon[53210]: pgmap v47: 97 pgs: 97 active+clean; 348 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 412 op/s 2024-02-10T06:51:47.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:47.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:47.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:51:48.642 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:51:48.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 206661 11.04 MB/sec execute 309 sec latency 41.553 ms 2024-02-10T06:51:48.738 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:51:49.650 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:49 smithi008 ceph-mon[66415]: pgmap v48: 97 pgs: 97 active+clean; 348 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.0 MiB/s wr, 410 op/s 2024-02-10T06:51:49.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 207077 11.05 MB/sec execute 310 sec latency 34.117 ms 2024-02-10T06:51:49.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:49 smithi029 ceph-mon[53210]: pgmap v48: 97 pgs: 97 active+clean; 348 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.0 MiB/s wr, 410 op/s 2024-02-10T06:51:50.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 207433 11.04 MB/sec execute 311 sec latency 46.695 ms 2024-02-10T06:51:50.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:50 smithi008 ceph-mon[66415]: mgrmap e41: smithi008.pxifbt(active, since 91s), standbys: smithi029.fuvrtf 2024-02-10T06:51:50.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:50 smithi029 ceph-mon[53210]: mgrmap e41: smithi008.pxifbt(active, since 91s), standbys: smithi029.fuvrtf 2024-02-10T06:51:51.656 INFO:teuthology.orchestra.run.smithi008.stdout: 5 207724 11.03 MB/sec execute 312 sec latency 121.634 ms 2024-02-10T06:51:51.981 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:51 smithi008 ceph-mon[66415]: pgmap v49: 97 pgs: 97 active+clean; 348 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.8 MiB/s wr, 667 op/s 2024-02-10T06:51:51.981 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:51.981 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:51 smithi029 ceph-mon[53210]: pgmap v49: 97 pgs: 97 active+clean; 348 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.8 MiB/s wr, 667 op/s 2024-02-10T06:51:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:52.657 INFO:teuthology.orchestra.run.smithi008.stdout: 5 208055 11.03 MB/sec execute 313 sec latency 42.792 ms 2024-02-10T06:51:53.324 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:53 smithi029 ceph-mon[53210]: pgmap v50: 97 pgs: 97 active+clean; 348 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 MiB/s wr, 504 op/s 2024-02-10T06:51:53.325 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:53 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:53.325 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:53 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:53.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:53 smithi008 ceph-mon[66415]: pgmap v50: 97 pgs: 97 active+clean; 348 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 MiB/s wr, 504 op/s 2024-02-10T06:51:53.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:53 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:53.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:53 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:53.657 INFO:teuthology.orchestra.run.smithi008.stdout: 5 208476 11.03 MB/sec execute 314 sec latency 41.445 ms 2024-02-10T06:51:54.657 INFO:teuthology.orchestra.run.smithi008.stdout: 5 208859 11.03 MB/sec execute 315 sec latency 48.210 ms 2024-02-10T06:51:55.657 INFO:teuthology.orchestra.run.smithi008.stdout: 5 209261 11.04 MB/sec execute 316 sec latency 38.380 ms 2024-02-10T06:51:55.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:55 smithi008 ceph-mon[66415]: pgmap v51: 97 pgs: 97 active+clean; 344 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 7.3 MiB/s wr, 724 op/s 2024-02-10T06:51:55.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:55 smithi029 ceph-mon[53210]: pgmap v51: 97 pgs: 97 active+clean; 344 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 7.3 MiB/s wr, 724 op/s 2024-02-10T06:51:56.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 209549 11.02 MB/sec execute 317 sec latency 97.789 ms 2024-02-10T06:51:57.382 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:57.382 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:57.382 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:57 smithi008 ceph-mon[66415]: pgmap v52: 97 pgs: 97 active+clean; 344 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 4.9 MiB/s wr, 479 op/s 2024-02-10T06:51:57.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:57.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:57.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:57 smithi029 ceph-mon[53210]: pgmap v52: 97 pgs: 97 active+clean; 344 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 4.9 MiB/s wr, 479 op/s 2024-02-10T06:51:57.657 INFO:teuthology.orchestra.run.smithi008.stdout: 5 209791 11.00 MB/sec execute 318 sec latency 208.081 ms 2024-02-10T06:51:58.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 210032 10.98 MB/sec execute 319 sec latency 294.886 ms 2024-02-10T06:51:58.735 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:58 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:58.735 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:58 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:58.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:58 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:58.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:58 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:51:59.539 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:51:59.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 210214 10.96 MB/sec execute 320 sec latency 69.972 ms 2024-02-10T06:52:00.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:59 smithi008 ceph-mon[66415]: pgmap v53: 97 pgs: 97 active+clean; 344 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.9 MiB/s wr, 478 op/s 2024-02-10T06:52:00.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:59 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:00.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:51:59 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:59 smithi029 ceph-mon[53210]: pgmap v53: 97 pgs: 97 active+clean; 344 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.9 MiB/s wr, 478 op/s 2024-02-10T06:52:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:59 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:51:59 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:00.366 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:52:00.366 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (4m) 4s ago 14m 15.5M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:52:00.366 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (20s) 4s ago 13m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T06:52:00.366 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (14s) 9s ago 13m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T06:52:00.366 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (3m) 4s ago 13m 43.2M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:52:00.366 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (7m) 4s ago 7m 38.1M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:52:00.367 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (7m) 9s ago 7m 14.5M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:52:00.367 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (3m) 4s ago 15m 496M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:52:00.367 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (2m) 9s ago 13m 449M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:52:00.367 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (112s) 4s ago 15m 44.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:52:00.367 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (82s) 9s ago 13m 35.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:52:00.367 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (7m) 4s ago 7m 277M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:52:00.367 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (7m) 9s ago 7m 84.3M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:52:00.367 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (3m) 4s ago 13m 9731k - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:52:00.367 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (3m) 9s ago 13m 18.7M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:52:00.367 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (12m) 4s ago 12m 433M 1937M 17.2.0 e1d6a67b021e 139b7a92bfc4 2024-02-10T06:52:00.367 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (12m) 4s ago 12m 351M 1937M 17.2.0 e1d6a67b021e 0f1119e17a52 2024-02-10T06:52:00.367 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (11m) 4s ago 11m 518M 1937M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:52:00.367 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (11m) 4s ago 11m 361M 1937M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:52:00.367 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (10m) 9s ago 10m 607M 2705M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:52:00.368 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (10m) 9s ago 10m 461M 2705M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:52:00.368 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (9m) 9s ago 9m 381M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:52:00.368 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (9m) 9s ago 9m 588M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:52:00.368 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (2m) 4s ago 12m 58.4M - 2.33.4 514e6a882f6e 193819b86dea 2024-02-10T06:52:00.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 210488 10.96 MB/sec execute 321 sec latency 56.021 ms 2024-02-10T06:52:00.807 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:00 smithi008 ceph-mon[66415]: from='client.34202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:52:00.807 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:00 smithi008 ceph-mon[66415]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:52:01.039 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:52:01.040 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:52:01.040 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:52:01.040 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:52:01.040 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:52:01.040 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:52:01.040 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:52:01.040 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:52:01.040 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T06:52:01.040 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:52:01.040 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:52:01.040 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:52:01.040 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:52:01.040 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:52:01.040 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-10T06:52:01.040 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 4 2024-02-10T06:52:01.041 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:52:01.041 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:52:01.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:00 smithi029 ceph-mon[53210]: from='client.34202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:52:01.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:00 smithi029 ceph-mon[53210]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:52:01.441 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:52:01.441 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:52:01.441 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:52:01.442 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:52:01.442 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:52:01.442 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:52:01.442 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-02-10T06:52:01.442 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-02-10T06:52:01.442 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:52:01.442 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "6/23 daemons upgraded", 2024-02-10T06:52:01.442 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading crash daemons", 2024-02-10T06:52:01.442 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:52:01.442 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:52:01.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 210758 10.95 MB/sec execute 322 sec latency 78.870 ms 2024-02-10T06:52:01.828 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:01 smithi008 ceph-mon[66415]: pgmap v54: 97 pgs: 97 active+clean; 346 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 657 op/s 2024-02-10T06:52:01.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:01 smithi008 ceph-mon[66415]: from='client.34210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:52:01.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:01 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/2989977834' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:52:01.949 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:52:02.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:01 smithi029 ceph-mon[53210]: pgmap v54: 97 pgs: 97 active+clean; 346 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 657 op/s 2024-02-10T06:52:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:01 smithi029 ceph-mon[53210]: from='client.34210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:52:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:01 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/2989977834' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:52:02.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 211012 10.93 MB/sec execute 323 sec latency 72.512 ms 2024-02-10T06:52:02.882 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:02 smithi008 ceph-mon[66415]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:52:02.882 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:02 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/3762108081' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:52:03.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:02 smithi029 ceph-mon[53210]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:52:03.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:02 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/3762108081' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:52:03.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 211271 10.92 MB/sec execute 324 sec latency 66.574 ms 2024-02-10T06:52:03.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: pgmap v55: 97 pgs: 97 active+clean; 346 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.0 MiB/s wr, 400 op/s 2024-02-10T06:52:03.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:52:03.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:03.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:03.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:52:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:52:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:52:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:52:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:52:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:52:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi008"}]: dispatch 2024-02-10T06:52:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi008"}]': finished 2024-02-10T06:52:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi029"}]: dispatch 2024-02-10T06:52:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi029"}]': finished 2024-02-10T06:52:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T06:52:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: pgmap v55: 97 pgs: 97 active+clean; 346 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.0 MiB/s wr, 400 op/s 2024-02-10T06:52:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:52:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:52:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:52:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:04.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:04.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:52:04.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:52:04.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:52:04.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:52:04.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:04.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi008"}]: dispatch 2024-02-10T06:52:04.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi008"}]': finished 2024-02-10T06:52:04.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi029"}]: dispatch 2024-02-10T06:52:04.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi029"}]': finished 2024-02-10T06:52:04.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T06:52:04.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 211490 10.90 MB/sec execute 325 sec latency 44.394 ms 2024-02-10T06:52:04.918 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:04 smithi008 ceph-mon[66415]: Upgrade: Setting container_image for all crash 2024-02-10T06:52:04.918 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:04 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T06:52:04.918 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:04 smithi008 ceph-mon[66415]: Upgrade: osd.0 is safe to restart 2024-02-10T06:52:05.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:04 smithi029 ceph-mon[53210]: Upgrade: Setting container_image for all crash 2024-02-10T06:52:05.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:04 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T06:52:05.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:04 smithi029 ceph-mon[53210]: Upgrade: osd.0 is safe to restart 2024-02-10T06:52:05.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 211786 10.90 MB/sec execute 326 sec latency 50.844 ms 2024-02-10T06:52:05.923 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:05 smithi008 ceph-mon[66415]: pgmap v56: 97 pgs: 97 active+clean; 344 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.5 MiB/s wr, 553 op/s 2024-02-10T06:52:06.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:05 smithi029 ceph-mon[53210]: pgmap v56: 97 pgs: 97 active+clean; 344 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.5 MiB/s wr, 553 op/s 2024-02-10T06:52:06.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 212071 10.88 MB/sec execute 327 sec latency 56.146 ms 2024-02-10T06:52:07.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:07 smithi008 ceph-mon[66415]: pgmap v57: 97 pgs: 97 active+clean; 344 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.3 MiB/s wr, 335 op/s 2024-02-10T06:52:07.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:07 smithi008 ceph-mon[66415]: Upgrade: Updating osd.0 2024-02-10T06:52:07.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:07 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:07.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:07 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T06:52:07.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:07 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:52:07.482 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:07 smithi008 ceph-mon[66415]: Deploying daemon osd.0 on smithi008 2024-02-10T06:52:07.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 212310 10.87 MB/sec execute 328 sec latency 51.732 ms 2024-02-10T06:52:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:07 smithi029 ceph-mon[53210]: pgmap v57: 97 pgs: 97 active+clean; 344 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.3 MiB/s wr, 335 op/s 2024-02-10T06:52:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:07 smithi029 ceph-mon[53210]: Upgrade: Updating osd.0 2024-02-10T06:52:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:07 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:07 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T06:52:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:07 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:52:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:07 smithi029 ceph-mon[53210]: Deploying daemon osd.0 on smithi008 2024-02-10T06:52:08.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 212659 10.88 MB/sec execute 329 sec latency 64.180 ms 2024-02-10T06:52:09.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 213044 10.87 MB/sec execute 330 sec latency 35.487 ms 2024-02-10T06:52:09.801 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:09 smithi008 ceph-mon[66415]: pgmap v58: 97 pgs: 97 active+clean; 344 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.3 MiB/s wr, 334 op/s 2024-02-10T06:52:09.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:09 smithi029 ceph-mon[53210]: pgmap v58: 97 pgs: 97 active+clean; 344 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.3 MiB/s wr, 334 op/s 2024-02-10T06:52:10.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 213444 10.86 MB/sec execute 331 sec latency 51.029 ms 2024-02-10T06:52:11.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 213707 10.85 MB/sec execute 332 sec latency 55.361 ms 2024-02-10T06:52:11.702 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:11 smithi008 ceph-mon[66415]: pgmap v59: 97 pgs: 97 active+clean; 347 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.2 MiB/s wr, 519 op/s 2024-02-10T06:52:11.956 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:11 smithi008 systemd[1]: Stopping Ceph osd.0 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:52:11.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:11 smithi029 ceph-mon[53210]: pgmap v59: 97 pgs: 97 active+clean; 347 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.2 MiB/s wr, 519 op/s 2024-02-10T06:52:12.230 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:12 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0[34782]: 2024-02-10T06:52:12.206+0000 7fdace73f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:52:12.230 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:12 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0[34782]: 2024-02-10T06:52:12.206+0000 7fdace73f700 -1 osd.0 73 *** Got signal Terminated *** 2024-02-10T06:52:12.230 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:12 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0[34782]: 2024-02-10T06:52:12.206+0000 7fdace73f700 -1 osd.0 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:52:12.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 214058 10.85 MB/sec execute 333 sec latency 53.387 ms 2024-02-10T06:52:12.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:12 smithi008 ceph-mon[66415]: osd.0 marked itself down and dead 2024-02-10T06:52:12.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:12 smithi029 ceph-mon[53210]: osd.0 marked itself down and dead 2024-02-10T06:52:13.591 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:13 smithi008 podman[72112]: 2024-02-10 06:52:13.309011971 +0000 UTC m=+1.239942787 container died 139b7a92bfc4412329b811fc08d5cee2eba494bb13f186e6268a2151fd0247b9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, io.buildah.version=1.19.8, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, ceph=True, com.redhat.component=centos-stream-container, release=754, name=centos-stream, io.openshift.tags=base centos centos-stream) 2024-02-10T06:52:13.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 214058 10.82 MB/sec execute 334 sec latency 1016.689 ms 2024-02-10T06:52:13.846 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:13 smithi008 podman[72112]: 2024-02-10 06:52:13.69814118 +0000 UTC m=+1.629071997 container cleanup 139b7a92bfc4412329b811fc08d5cee2eba494bb13f186e6268a2151fd0247b9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0, ceph=True, name=centos-stream, version=8, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, release=754, vcs-type=git, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=) 2024-02-10T06:52:13.846 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:13 smithi008 bash[72112]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0 2024-02-10T06:52:13.846 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:13 smithi008 ceph-mon[66415]: pgmap v60: 97 pgs: 97 active+clean; 347 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.4 MiB/s wr, 340 op/s 2024-02-10T06:52:13.846 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:13 smithi008 ceph-mon[66415]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:52:13.847 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:13 smithi008 ceph-mon[66415]: failed to encode map e74 with expected crc 2024-02-10T06:52:13.847 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:13 smithi008 ceph-mon[66415]: osdmap e74: 8 total, 7 up, 8 in 2024-02-10T06:52:13.847 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:13 smithi008 ceph-mon[66415]: failed to encode map e74 with expected crc 2024-02-10T06:52:13.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:13 smithi029 ceph-mon[53210]: pgmap v60: 97 pgs: 97 active+clean; 347 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.4 MiB/s wr, 340 op/s 2024-02-10T06:52:13.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:13 smithi029 ceph-mon[53210]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:52:13.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:13 smithi029 ceph-mon[53210]: failed to encode map e74 with expected crc 2024-02-10T06:52:13.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:13 smithi029 ceph-mon[53210]: osdmap e74: 8 total, 7 up, 8 in 2024-02-10T06:52:13.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:13 smithi029 ceph-mon[53210]: failed to encode map e74 with expected crc 2024-02-10T06:52:14.101 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:13 smithi008 podman[72128]: 2024-02-10 06:52:13.996491766 +0000 UTC m=+0.690120842 container remove 139b7a92bfc4412329b811fc08d5cee2eba494bb13f186e6268a2151fd0247b9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, version=8, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, GIT_CLEAN=True, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vendor=Red Hat, Inc., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, ceph=True) 2024-02-10T06:52:14.603 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:14 smithi008 podman[72211]: 2024-02-10 06:52:14.409541078 +0000 UTC m=+0.022545487 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:52:14.603 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:14 smithi008 podman[72211]: 2024-02-10 06:52:14.549950633 +0000 UTC m=+0.162955028 container create f5629c512f915fd6c27c60c8588fe65fbfc1ca6a3d3d9fd2d0a776e9a3c9e5e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-290212a) 2024-02-10T06:52:14.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 214438 10.82 MB/sec execute 335 sec latency 1038.307 ms 2024-02-10T06:52:14.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:14 smithi008 ceph-mon[66415]: failed to encode map e75 with expected crc 2024-02-10T06:52:14.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:14 smithi008 ceph-mon[66415]: osdmap e75: 8 total, 7 up, 8 in 2024-02-10T06:52:14.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:14 smithi008 ceph-mon[66415]: failed to encode map e75 with expected crc 2024-02-10T06:52:14.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:14 smithi008 ceph-mon[66415]: failed to encode map e75 with expected crc 2024-02-10T06:52:14.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:14 smithi008 ceph-mon[66415]: failed to encode map e75 with expected crc 2024-02-10T06:52:14.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:14 smithi008 ceph-mon[66415]: failed to encode map e75 with expected crc 2024-02-10T06:52:14.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:14 smithi008 ceph-mon[66415]: failed to encode map e75 with expected crc 2024-02-10T06:52:14.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:14 smithi008 ceph-mon[66415]: failed to encode map e75 with expected crc 2024-02-10T06:52:14.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:14 smithi008 ceph-mon[66415]: failed to encode map e75 with expected crc 2024-02-10T06:52:14.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:14 smithi029 ceph-mon[53210]: failed to encode map e75 with expected crc 2024-02-10T06:52:14.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:14 smithi029 ceph-mon[53210]: osdmap e75: 8 total, 7 up, 8 in 2024-02-10T06:52:14.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:14 smithi029 ceph-mon[53210]: failed to encode map e75 with expected crc 2024-02-10T06:52:14.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:14 smithi029 ceph-mon[53210]: failed to encode map e75 with expected crc 2024-02-10T06:52:14.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:14 smithi029 ceph-mon[53210]: failed to encode map e75 with expected crc 2024-02-10T06:52:14.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:14 smithi029 ceph-mon[53210]: failed to encode map e75 with expected crc 2024-02-10T06:52:14.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:14 smithi029 ceph-mon[53210]: failed to encode map e75 with expected crc 2024-02-10T06:52:14.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:14 smithi029 ceph-mon[53210]: failed to encode map e75 with expected crc 2024-02-10T06:52:14.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:14 smithi029 ceph-mon[53210]: failed to encode map e75 with expected crc 2024-02-10T06:52:15.112 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:14 smithi008 podman[72211]: 2024-02-10 06:52:14.861490976 +0000 UTC m=+0.474495388 container init f5629c512f915fd6c27c60c8588fe65fbfc1ca6a3d3d9fd2d0a776e9a3c9e5e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-10T06:52:15.112 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:14 smithi008 podman[72211]: 2024-02-10 06:52:14.867521921 +0000 UTC m=+0.480526321 container start f5629c512f915fd6c27c60c8588fe65fbfc1ca6a3d3d9fd2d0a776e9a3c9e5e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, RELEASE=main-290212a, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-10T06:52:15.112 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:15 smithi008 podman[72211]: 2024-02-10 06:52:15.038038927 +0000 UTC m=+0.651043341 container attach f5629c512f915fd6c27c60c8588fe65fbfc1ca6a3d3d9fd2d0a776e9a3c9e5e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS) 2024-02-10T06:52:15.363 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:15 smithi008 podman[72211]: 2024-02-10 06:52:15.314221679 +0000 UTC m=+0.927226081 container died f5629c512f915fd6c27c60c8588fe65fbfc1ca6a3d3d9fd2d0a776e9a3c9e5e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2) 2024-02-10T06:52:15.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 214811 10.81 MB/sec execute 336 sec latency 42.542 ms 2024-02-10T06:52:15.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:15 smithi008 ceph-mon[66415]: pgmap v63: 97 pgs: 26 active+undersized+degraded, 7 active+undersized, 64 active+clean; 347 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 4.1 KiB/s rd, 5.3 MiB/s wr, 532 op/s; 197/1557 objects degraded (12.653%) 2024-02-10T06:52:15.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:15 smithi008 ceph-mon[66415]: Health check failed: Degraded data redundancy: 197/1557 objects degraded (12.653%), 26 pgs degraded (PG_DEGRADED) 2024-02-10T06:52:15.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:15 smithi029 ceph-mon[53210]: pgmap v63: 97 pgs: 26 active+undersized+degraded, 7 active+undersized, 64 active+clean; 347 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 4.1 KiB/s rd, 5.3 MiB/s wr, 532 op/s; 197/1557 objects degraded (12.653%) 2024-02-10T06:52:15.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:15 smithi029 ceph-mon[53210]: Health check failed: Degraded data redundancy: 197/1557 objects degraded (12.653%), 26 pgs degraded (PG_DEGRADED) 2024-02-10T06:52:16.640 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:16 smithi008 podman[72235]: 2024-02-10 06:52:16.57108487 +0000 UTC m=+1.246474097 container remove f5629c512f915fd6c27c60c8588fe65fbfc1ca6a3d3d9fd2d0a776e9a3c9e5e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-10T06:52:16.640 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:16 smithi008 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-10T06:52:16.640 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:16 smithi008 systemd[1]: Stopped Ceph osd.0 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:52:16.641 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:16 smithi008 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.0.service: Consumed 3min 50.381s CPU time. 2024-02-10T06:52:16.658 INFO:teuthology.orchestra.run.smithi008.stdout: 5 215148 10.81 MB/sec execute 337 sec latency 38.246 ms 2024-02-10T06:52:17.147 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:16 smithi008 systemd[1]: Starting Ceph osd.0 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:52:17.399 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:17 smithi008 podman[72331]: 2024-02-10 06:52:17.077602221 +0000 UTC m=+0.022474263 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:52:17.399 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:17 smithi008 podman[72331]: 2024-02-10 06:52:17.244695916 +0000 UTC m=+0.189567960 container create 60d490e8c9de0ee798b95a7f1151157c6d27e431037396bfc22e727bbc13e7a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-10T06:52:17.659 INFO:teuthology.orchestra.run.smithi008.stdout: 5 215556 10.80 MB/sec execute 338 sec latency 49.275 ms 2024-02-10T06:52:17.914 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:17 smithi008 podman[72331]: 2024-02-10 06:52:17.722050014 +0000 UTC m=+0.666922058 container init 60d490e8c9de0ee798b95a7f1151157c6d27e431037396bfc22e727bbc13e7a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, ceph=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-02-10T06:52:17.914 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:17 smithi008 podman[72331]: 2024-02-10 06:52:17.73271741 +0000 UTC m=+0.677589457 container start 60d490e8c9de0ee798b95a7f1151157c6d27e431037396bfc22e727bbc13e7a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.vendor=CentOS) 2024-02-10T06:52:17.914 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:17 smithi008 podman[72331]: 2024-02-10 06:52:17.897380563 +0000 UTC m=+0.842252602 container attach 60d490e8c9de0ee798b95a7f1151157c6d27e431037396bfc22e727bbc13e7a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-activate, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-10T06:52:17.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:17 smithi008 ceph-mon[66415]: pgmap v64: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 347 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.3 MiB/s wr, 534 op/s; 218/1557 objects degraded (14.001%) 2024-02-10T06:52:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:17 smithi029 ceph-mon[53210]: pgmap v64: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 347 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.3 MiB/s wr, 534 op/s; 218/1557 objects degraded (14.001%) 2024-02-10T06:52:18.421 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:18 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-activate[72345]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T06:52:18.422 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:18 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-activate[72345]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-10T06:52:18.422 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:18 smithi008 bash[72331]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T06:52:18.422 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:18 smithi008 bash[72331]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-10T06:52:18.422 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:18 smithi008 bash[72331]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-10T06:52:18.422 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:18 smithi008 bash[72331]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T06:52:18.422 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:18 smithi008 bash[72331]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-10T06:52:18.422 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:18 smithi008 bash[72331]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T06:52:18.422 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:18 smithi008 bash[72331]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-10T06:52:18.422 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:18 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-activate[72345]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-10T06:52:18.422 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:18 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-activate[72345]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T06:52:18.422 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:18 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-activate[72345]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-10T06:52:18.422 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:18 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-activate[72345]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T06:52:18.422 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:18 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-activate[72345]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-10T06:52:18.659 INFO:teuthology.orchestra.run.smithi008.stdout: 5 215932 10.80 MB/sec execute 339 sec latency 65.290 ms 2024-02-10T06:52:18.673 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:18 smithi008 podman[72467]: 2024-02-10 06:52:18.421895461 +0000 UTC m=+0.028050086 container died 60d490e8c9de0ee798b95a7f1151157c6d27e431037396bfc22e727bbc13e7a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-activate, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-290212a) 2024-02-10T06:52:19.428 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:19 smithi008 ceph-mon[66415]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 347 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 2.5 MiB/s wr, 255 op/s; 218/1557 objects degraded (14.001%) 2024-02-10T06:52:19.428 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:19 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:19.428 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:19 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:52:19.659 INFO:teuthology.orchestra.run.smithi008.stdout: 5 216464 10.82 MB/sec execute 340 sec latency 36.369 ms 2024-02-10T06:52:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:19 smithi029 ceph-mon[53210]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 347 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 2.5 MiB/s wr, 255 op/s; 218/1557 objects degraded (14.001%) 2024-02-10T06:52:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:19 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:19 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:52:19.940 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:19 smithi008 podman[72467]: 2024-02-10 06:52:19.717653138 +0000 UTC m=+1.323807767 container remove 60d490e8c9de0ee798b95a7f1151157c6d27e431037396bfc22e727bbc13e7a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, RELEASE=main-290212a, maintainer=Guillaume Abrioux ) 2024-02-10T06:52:20.194 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:19 smithi008 podman[72523]: 2024-02-10 06:52:19.866018777 +0000 UTC m=+0.020993595 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:52:20.194 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:20 smithi008 podman[72523]: 2024-02-10 06:52:20.182974234 +0000 UTC m=+0.337949038 container create acb0107eae6cd46a6db146051796c3e47490b8c69574663279ec5aa184d5440f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-10T06:52:20.460 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:20 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 230/1689 objects degraded (13.618%), 29 pgs degraded (PG_DEGRADED) 2024-02-10T06:52:20.659 INFO:teuthology.orchestra.run.smithi008.stdout: 5 216824 10.81 MB/sec execute 341 sec latency 44.708 ms 2024-02-10T06:52:20.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:20 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 230/1689 objects degraded (13.618%), 29 pgs degraded (PG_DEGRADED) 2024-02-10T06:52:20.711 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:20 smithi008 podman[72523]: 2024-02-10 06:52:20.620817443 +0000 UTC m=+0.775792255 container init acb0107eae6cd46a6db146051796c3e47490b8c69574663279ec5aa184d5440f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, RELEASE=main-290212a) 2024-02-10T06:52:20.711 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:20 smithi008 podman[72523]: 2024-02-10 06:52:20.638239726 +0000 UTC m=+0.793214533 container start acb0107eae6cd46a6db146051796c3e47490b8c69574663279ec5aa184d5440f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-10T06:52:20.711 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:20 smithi008 ceph-osd[72541]: -- 172.21.15.8:0/950027003 <== mon.0 v2:172.21.15.8:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560e83eef1e0 con 0x560e83ec0c00 2024-02-10T06:52:20.962 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:20 smithi008 bash[72523]: acb0107eae6cd46a6db146051796c3e47490b8c69574663279ec5aa184d5440f 2024-02-10T06:52:20.962 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:20 smithi008 systemd[1]: Started Ceph osd.0 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:52:21.468 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:21 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0[72537]: 2024-02-10T06:52:21.229+0000 7fda891607c0 -1 Falling back to public interface 2024-02-10T06:52:21.665 INFO:teuthology.orchestra.run.smithi008.stdout: 5 217135 10.80 MB/sec execute 342 sec latency 39.269 ms 2024-02-10T06:52:21.973 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:21 smithi008 ceph-mon[66415]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 349 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.2 MiB/s wr, 611 op/s; 230/1689 objects degraded (13.618%) 2024-02-10T06:52:21.973 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:21 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:21.973 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:21 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:21.973 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:21 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:52:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:21 smithi029 ceph-mon[53210]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 349 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.2 MiB/s wr, 611 op/s; 230/1689 objects degraded (13.618%) 2024-02-10T06:52:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:21 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:21 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:21 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:52:22.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 217546 10.81 MB/sec execute 343 sec latency 36.390 ms 2024-02-10T06:52:23.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:23 smithi008 ceph-mon[66415]: pgmap v67: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 349 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.1 MiB/s wr, 508 op/s; 230/1689 objects degraded (13.618%) 2024-02-10T06:52:23.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:23 smithi029 ceph-mon[53210]: pgmap v67: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 349 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.1 MiB/s wr, 508 op/s; 230/1689 objects degraded (13.618%) 2024-02-10T06:52:23.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 217952 10.81 MB/sec execute 344 sec latency 61.228 ms 2024-02-10T06:52:24.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 218388 10.81 MB/sec execute 345 sec latency 41.116 ms 2024-02-10T06:52:25.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 218770 10.81 MB/sec execute 346 sec latency 31.212 ms 2024-02-10T06:52:25.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:25 smithi008 ceph-mon[66415]: pgmap v68: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 350 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 7.3 MiB/s wr, 726 op/s; 230/1686 objects degraded (13.642%) 2024-02-10T06:52:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:25 smithi029 ceph-mon[53210]: pgmap v68: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 350 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 7.3 MiB/s wr, 726 op/s; 230/1686 objects degraded (13.642%) 2024-02-10T06:52:26.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 219179 10.80 MB/sec execute 347 sec latency 33.650 ms 2024-02-10T06:52:26.794 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:26 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 229/1683 objects degraded (13.607%), 29 pgs degraded (PG_DEGRADED) 2024-02-10T06:52:26.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:26 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 229/1683 objects degraded (13.607%), 29 pgs degraded (PG_DEGRADED) 2024-02-10T06:52:27.300 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:27 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0[72537]: 2024-02-10T06:52:27.153+0000 7fda891607c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-02-10T06:52:27.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 219494 10.80 MB/sec execute 348 sec latency 51.653 ms 2024-02-10T06:52:27.812 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:27 smithi008 ceph-mon[66415]: pgmap v69: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.8 MiB/s wr, 475 op/s; 229/1683 objects degraded (13.607%) 2024-02-10T06:52:27.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:27 smithi029 ceph-mon[53210]: pgmap v69: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.8 MiB/s wr, 475 op/s; 229/1683 objects degraded (13.607%) 2024-02-10T06:52:28.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 219851 10.81 MB/sec execute 349 sec latency 47.962 ms 2024-02-10T06:52:28.817 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:28 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0[72537]: 2024-02-10T06:52:28.792+0000 7fda891607c0 -1 osd.0 73 log_to_monitors true 2024-02-10T06:52:29.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 220249 10.80 MB/sec execute 350 sec latency 117.025 ms 2024-02-10T06:52:29.828 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:29 smithi008 ceph-mon[66415]: pgmap v70: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 474 op/s; 229/1683 objects degraded (13.607%) 2024-02-10T06:52:29.828 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:29 smithi008 ceph-mon[66415]: from='osd.0 [v2:172.21.15.8:6802/2399431344,v1:172.21.15.8:6803/2399431344]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T06:52:29.829 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 06:52:29 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0[72537]: 2024-02-10T06:52:29.754+0000 7fda7a5b8700 -1 osd.0 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:52:29.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:29 smithi029 ceph-mon[53210]: pgmap v70: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 474 op/s; 229/1683 objects degraded (13.607%) 2024-02-10T06:52:29.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:29 smithi029 ceph-mon[53210]: from='osd.0 [v2:172.21.15.8:6802/2399431344,v1:172.21.15.8:6803/2399431344]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T06:52:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 220660 10.79 MB/sec execute 351 sec latency 31.586 ms 2024-02-10T06:52:30.849 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:30 smithi008 ceph-mon[66415]: from='osd.0 [v2:172.21.15.8:6802/2399431344,v1:172.21.15.8:6803/2399431344]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T06:52:30.849 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:30 smithi008 ceph-mon[66415]: failed to encode map e76 with expected crc 2024-02-10T06:52:30.849 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:30 smithi008 ceph-mon[66415]: osdmap e76: 8 total, 7 up, 8 in 2024-02-10T06:52:30.849 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:30 smithi008 ceph-mon[66415]: from='osd.0 [v2:172.21.15.8:6802/2399431344,v1:172.21.15.8:6803/2399431344]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T06:52:30.849 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:30 smithi008 ceph-mon[66415]: failed to encode map e76 with expected crc 2024-02-10T06:52:30.849 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:30 smithi008 ceph-mon[66415]: failed to encode map e76 with expected crc 2024-02-10T06:52:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:30 smithi029 ceph-mon[53210]: from='osd.0 [v2:172.21.15.8:6802/2399431344,v1:172.21.15.8:6803/2399431344]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T06:52:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:30 smithi029 ceph-mon[53210]: failed to encode map e76 with expected crc 2024-02-10T06:52:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:30 smithi029 ceph-mon[53210]: osdmap e76: 8 total, 7 up, 8 in 2024-02-10T06:52:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:30 smithi029 ceph-mon[53210]: from='osd.0 [v2:172.21.15.8:6802/2399431344,v1:172.21.15.8:6803/2399431344]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T06:52:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:30 smithi029 ceph-mon[53210]: failed to encode map e76 with expected crc 2024-02-10T06:52:31.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:30 smithi029 ceph-mon[53210]: failed to encode map e76 with expected crc 2024-02-10T06:52:31.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 220826 10.77 MB/sec execute 352 sec latency 828.735 ms 2024-02-10T06:52:32.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:31 smithi008 ceph-mon[66415]: pgmap v72: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 352 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.5 MiB/s wr, 564 op/s; 223/1638 objects degraded (13.614%) 2024-02-10T06:52:32.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:31 smithi008 ceph-mon[66415]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:52:32.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:31 smithi008 ceph-mon[66415]: failed to encode map e77 with expected crc 2024-02-10T06:52:32.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:31 smithi008 ceph-mon[66415]: osd.0 [v2:172.21.15.8:6802/2399431344,v1:172.21.15.8:6803/2399431344] boot 2024-02-10T06:52:32.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:31 smithi008 ceph-mon[66415]: osdmap e77: 8 total, 8 up, 8 in 2024-02-10T06:52:32.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:52:32.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:31 smithi008 ceph-mon[66415]: failed to encode map e77 with expected crc 2024-02-10T06:52:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:31 smithi029 ceph-mon[53210]: pgmap v72: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 352 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.5 MiB/s wr, 564 op/s; 223/1638 objects degraded (13.614%) 2024-02-10T06:52:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:31 smithi029 ceph-mon[53210]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:52:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:31 smithi029 ceph-mon[53210]: failed to encode map e77 with expected crc 2024-02-10T06:52:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:31 smithi029 ceph-mon[53210]: osd.0 [v2:172.21.15.8:6802/2399431344,v1:172.21.15.8:6803/2399431344] boot 2024-02-10T06:52:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:31 smithi029 ceph-mon[53210]: osdmap e77: 8 total, 8 up, 8 in 2024-02-10T06:52:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:52:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:31 smithi029 ceph-mon[53210]: failed to encode map e77 with expected crc 2024-02-10T06:52:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:32.350 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:52:32.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 220830 10.74 MB/sec execute 353 sec latency 1828.767 ms 2024-02-10T06:52:32.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:32 smithi008 ceph-mon[66415]: failed to encode map e78 with expected crc 2024-02-10T06:52:32.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:32 smithi008 ceph-mon[66415]: osdmap e78: 8 total, 8 up, 8 in 2024-02-10T06:52:32.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:32 smithi008 ceph-mon[66415]: failed to encode map e78 with expected crc 2024-02-10T06:52:32.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:32 smithi008 ceph-mon[66415]: failed to encode map e78 with expected crc 2024-02-10T06:52:32.990 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:32 smithi008 ceph-mon[66415]: failed to encode map e78 with expected crc 2024-02-10T06:52:32.990 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:32 smithi008 ceph-mon[66415]: failed to encode map e78 with expected crc 2024-02-10T06:52:32.990 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:32 smithi008 ceph-mon[66415]: failed to encode map e78 with expected crc 2024-02-10T06:52:32.990 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:32 smithi008 ceph-mon[66415]: failed to encode map e78 with expected crc 2024-02-10T06:52:32.990 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:32.990 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:32 smithi029 ceph-mon[53210]: failed to encode map e78 with expected crc 2024-02-10T06:52:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:32 smithi029 ceph-mon[53210]: osdmap e78: 8 total, 8 up, 8 in 2024-02-10T06:52:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:32 smithi029 ceph-mon[53210]: failed to encode map e78 with expected crc 2024-02-10T06:52:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:32 smithi029 ceph-mon[53210]: failed to encode map e78 with expected crc 2024-02-10T06:52:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:32 smithi029 ceph-mon[53210]: failed to encode map e78 with expected crc 2024-02-10T06:52:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:32 smithi029 ceph-mon[53210]: failed to encode map e78 with expected crc 2024-02-10T06:52:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:32 smithi029 ceph-mon[53210]: failed to encode map e78 with expected crc 2024-02-10T06:52:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:32 smithi029 ceph-mon[53210]: failed to encode map e78 with expected crc 2024-02-10T06:52:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:33.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:33.281 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:52:33.281 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (4m) 2s ago 14m 15.5M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:52:33.281 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (53s) 2s ago 14m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T06:52:33.281 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (47s) 42s ago 13m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T06:52:33.281 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (4m) 2s ago 14m 43.8M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:52:33.281 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (8m) 2s ago 8m 38.1M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:52:33.281 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (8m) 42s ago 8m 14.5M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:52:33.281 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (3m) 2s ago 15m 499M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:52:33.282 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (2m) 42s ago 13m 449M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:52:33.282 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (2m) 2s ago 16m 47.3M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:52:33.282 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (115s) 42s ago 13m 35.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:52:33.282 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (8m) 2s ago 8m 269M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:52:33.282 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (8m) 42s ago 8m 84.3M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:52:33.282 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (4m) 2s ago 14m 11.7M - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:52:33.282 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (3m) 42s ago 13m 18.7M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:52:33.282 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (13s) 2s ago 13m 13.7M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 acb0107eae6c 2024-02-10T06:52:33.282 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (12m) 2s ago 12m 373M 1937M 17.2.0 e1d6a67b021e 0f1119e17a52 2024-02-10T06:52:33.282 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (12m) 2s ago 12m 554M 1937M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:52:33.282 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (11m) 2s ago 11m 382M 1937M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:52:33.282 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (11m) 42s ago 11m 607M 2705M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:52:33.282 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (10m) 42s ago 10m 461M 2705M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:52:33.283 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (10m) 42s ago 10m 381M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:52:33.283 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (9m) 42s ago 9m 588M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:52:33.283 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (2m) 2s ago 13m 58.5M - 2.33.4 514e6a882f6e 193819b86dea 2024-02-10T06:52:33.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 221109 10.74 MB/sec execute 354 sec latency 2063.305 ms 2024-02-10T06:52:33.745 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:33 smithi008 ceph-mon[66415]: pgmap v75: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 352 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 465 op/s; 223/1638 objects degraded (13.614%) 2024-02-10T06:52:33.745 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:33 smithi008 ceph-mon[66415]: from='client.34230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:52:33.745 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:33 smithi008 ceph-mon[66415]: from='client.44153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:52:33.745 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:33 smithi008 ceph-mon[66415]: failed to encode map e79 with expected crc 2024-02-10T06:52:33.745 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:33 smithi008 ceph-mon[66415]: osdmap e79: 8 total, 8 up, 8 in 2024-02-10T06:52:33.746 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:33 smithi008 ceph-mon[66415]: failed to encode map e79 with expected crc 2024-02-10T06:52:33.746 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:33 smithi008 ceph-mon[66415]: failed to encode map e79 with expected crc 2024-02-10T06:52:33.746 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:33 smithi008 ceph-mon[66415]: failed to encode map e79 with expected crc 2024-02-10T06:52:33.746 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:33 smithi008 ceph-mon[66415]: failed to encode map e79 with expected crc 2024-02-10T06:52:33.746 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:33 smithi008 ceph-mon[66415]: failed to encode map e79 with expected crc 2024-02-10T06:52:33.746 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:33 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:33.746 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:33 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:52:33.746 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:33 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 223/1638 objects degraded (13.614%), 29 pgs degraded (PG_DEGRADED) 2024-02-10T06:52:33.746 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:33 smithi008 ceph-mon[66415]: osdmap e80: 8 total, 8 up, 8 in 2024-02-10T06:52:33.910 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:52:33.910 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:52:33.910 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:52:33.910 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:52:33.910 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:52:33.910 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:52:33.911 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:52:33.911 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:52:33.911 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-10T06:52:33.911 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T06:52:33.911 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:52:33.911 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:52:33.911 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:52:33.911 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:52:33.911 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:52:33.911 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-02-10T06:52:33.911 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 5 2024-02-10T06:52:33.911 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:52:33.911 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:52:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:33 smithi029 ceph-mon[53210]: pgmap v75: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 352 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 465 op/s; 223/1638 objects degraded (13.614%) 2024-02-10T06:52:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:33 smithi029 ceph-mon[53210]: from='client.34230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:52:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:33 smithi029 ceph-mon[53210]: from='client.44153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:52:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:33 smithi029 ceph-mon[53210]: failed to encode map e79 with expected crc 2024-02-10T06:52:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:33 smithi029 ceph-mon[53210]: osdmap e79: 8 total, 8 up, 8 in 2024-02-10T06:52:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:33 smithi029 ceph-mon[53210]: failed to encode map e79 with expected crc 2024-02-10T06:52:34.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:33 smithi029 ceph-mon[53210]: failed to encode map e79 with expected crc 2024-02-10T06:52:34.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:33 smithi029 ceph-mon[53210]: failed to encode map e79 with expected crc 2024-02-10T06:52:34.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:33 smithi029 ceph-mon[53210]: failed to encode map e79 with expected crc 2024-02-10T06:52:34.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:33 smithi029 ceph-mon[53210]: failed to encode map e79 with expected crc 2024-02-10T06:52:34.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:33 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:34.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:33 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:52:34.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:33 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 223/1638 objects degraded (13.614%), 29 pgs degraded (PG_DEGRADED) 2024-02-10T06:52:34.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:33 smithi029 ceph-mon[53210]: osdmap e80: 8 total, 8 up, 8 in 2024-02-10T06:52:34.405 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:52:34.405 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:52:34.405 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:52:34.405 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:52:34.406 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:52:34.406 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:52:34.406 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-02-10T06:52:34.406 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-02-10T06:52:34.406 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:52:34.406 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "7/23 daemons upgraded", 2024-02-10T06:52:34.406 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:52:34.406 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:52:34.406 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:52:34.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 221440 10.75 MB/sec execute 355 sec latency 51.861 ms 2024-02-10T06:52:34.767 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:34 smithi008 ceph-mon[66415]: from='client.34238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:52:34.767 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:34 smithi008 ceph-mon[66415]: failed to encode map e80 with expected crc 2024-02-10T06:52:34.767 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:34 smithi008 ceph-mon[66415]: failed to encode map e80 with expected crc 2024-02-10T06:52:34.767 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:34 smithi008 ceph-mon[66415]: failed to encode map e80 with expected crc 2024-02-10T06:52:34.767 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:34 smithi008 ceph-mon[66415]: failed to encode map e80 with expected crc 2024-02-10T06:52:34.767 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:34 smithi008 ceph-mon[66415]: failed to encode map e80 with expected crc 2024-02-10T06:52:34.768 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:34 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/2585132816' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:52:34.768 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:34 smithi008 ceph-mon[66415]: osdmap e81: 8 total, 8 up, 8 in 2024-02-10T06:52:34.999 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN Degraded data redundancy: 106/1500 objects degraded (7.067%), 7 pgs degraded 2024-02-10T06:52:34.999 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 106/1500 objects degraded (7.067%), 7 pgs degraded 2024-02-10T06:52:35.000 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.13 is active+undersized+degraded, acting [4,3] 2024-02-10T06:52:35.000 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1f is active+undersized+degraded, acting [7,4] 2024-02-10T06:52:35.000 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1 is active+recovery_wait+undersized+degraded+remapped, acting [3,4] 2024-02-10T06:52:35.000 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.7 is active+recovery_wait+undersized+degraded+remapped, acting [3,7] 2024-02-10T06:52:35.000 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.b is active+recovery_wait+undersized+degraded+remapped, acting [3,4] 2024-02-10T06:52:35.000 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.f is active+recovery_wait+undersized+degraded+remapped, acting [7,4] 2024-02-10T06:52:35.000 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-02-10T06:52:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:34 smithi029 ceph-mon[53210]: from='client.34238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:52:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:34 smithi029 ceph-mon[53210]: failed to encode map e80 with expected crc 2024-02-10T06:52:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:34 smithi029 ceph-mon[53210]: failed to encode map e80 with expected crc 2024-02-10T06:52:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:34 smithi029 ceph-mon[53210]: failed to encode map e80 with expected crc 2024-02-10T06:52:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:34 smithi029 ceph-mon[53210]: failed to encode map e80 with expected crc 2024-02-10T06:52:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:34 smithi029 ceph-mon[53210]: failed to encode map e80 with expected crc 2024-02-10T06:52:35.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:34 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/2585132816' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:52:35.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:34 smithi029 ceph-mon[53210]: osdmap e81: 8 total, 8 up, 8 in 2024-02-10T06:52:35.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 221708 10.74 MB/sec execute 356 sec latency 849.775 ms 2024-02-10T06:52:35.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:35 smithi008 ceph-mon[66415]: pgmap v78: 97 pgs: 4 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 peering, 3 active+undersized+degraded, 5 active+undersized, 78 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.9 MiB/s wr, 480 op/s; 106/1500 objects degraded (7.067%); 2.4 MiB/s, 38 objects/s recovering 2024-02-10T06:52:35.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:35 smithi008 ceph-mon[66415]: from='client.44159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:52:35.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:35 smithi008 ceph-mon[66415]: failed to encode map e81 with expected crc 2024-02-10T06:52:35.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:35 smithi008 ceph-mon[66415]: failed to encode map e81 with expected crc 2024-02-10T06:52:35.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:35 smithi008 ceph-mon[66415]: failed to encode map e81 with expected crc 2024-02-10T06:52:35.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:35 smithi008 ceph-mon[66415]: failed to encode map e81 with expected crc 2024-02-10T06:52:35.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:35 smithi008 ceph-mon[66415]: failed to encode map e81 with expected crc 2024-02-10T06:52:35.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:35 smithi008 ceph-mon[66415]: failed to encode map e81 with expected crc 2024-02-10T06:52:35.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:35 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/2347942909' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:52:35.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:35 smithi008 ceph-mon[66415]: osdmap e82: 8 total, 8 up, 8 in 2024-02-10T06:52:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:35 smithi029 ceph-mon[53210]: pgmap v78: 97 pgs: 4 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 peering, 3 active+undersized+degraded, 5 active+undersized, 78 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.9 MiB/s wr, 480 op/s; 106/1500 objects degraded (7.067%); 2.4 MiB/s, 38 objects/s recovering 2024-02-10T06:52:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:35 smithi029 ceph-mon[53210]: from='client.44159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:52:36.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:35 smithi029 ceph-mon[53210]: failed to encode map e81 with expected crc 2024-02-10T06:52:36.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:35 smithi029 ceph-mon[53210]: failed to encode map e81 with expected crc 2024-02-10T06:52:36.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:35 smithi029 ceph-mon[53210]: failed to encode map e81 with expected crc 2024-02-10T06:52:36.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:35 smithi029 ceph-mon[53210]: failed to encode map e81 with expected crc 2024-02-10T06:52:36.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:35 smithi029 ceph-mon[53210]: failed to encode map e81 with expected crc 2024-02-10T06:52:36.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:35 smithi029 ceph-mon[53210]: failed to encode map e81 with expected crc 2024-02-10T06:52:36.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:35 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/2347942909' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:52:36.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:35 smithi029 ceph-mon[53210]: osdmap e82: 8 total, 8 up, 8 in 2024-02-10T06:52:36.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 221999 10.72 MB/sec execute 357 sec latency 880.230 ms 2024-02-10T06:52:36.809 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:36 smithi008 ceph-mon[66415]: failed to encode map e82 with expected crc 2024-02-10T06:52:36.809 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:36 smithi008 ceph-mon[66415]: failed to encode map e82 with expected crc 2024-02-10T06:52:36.809 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:36 smithi008 ceph-mon[66415]: failed to encode map e82 with expected crc 2024-02-10T06:52:36.809 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:36 smithi008 ceph-mon[66415]: failed to encode map e82 with expected crc 2024-02-10T06:52:36.809 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:36 smithi008 ceph-mon[66415]: failed to encode map e82 with expected crc 2024-02-10T06:52:36.809 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:36 smithi008 ceph-mon[66415]: failed to encode map e82 with expected crc 2024-02-10T06:52:36.810 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:36 smithi008 ceph-mon[66415]: failed to encode map e82 with expected crc 2024-02-10T06:52:36.810 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:36 smithi008 ceph-mon[66415]: failed to encode map e82 with expected crc 2024-02-10T06:52:36.810 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:36 smithi008 ceph-mon[66415]: failed to encode map e82 with expected crc 2024-02-10T06:52:36.810 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:36 smithi008 ceph-mon[66415]: osdmap e83: 8 total, 8 up, 8 in 2024-02-10T06:52:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:36 smithi029 ceph-mon[53210]: failed to encode map e82 with expected crc 2024-02-10T06:52:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:36 smithi029 ceph-mon[53210]: failed to encode map e82 with expected crc 2024-02-10T06:52:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:36 smithi029 ceph-mon[53210]: failed to encode map e82 with expected crc 2024-02-10T06:52:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:36 smithi029 ceph-mon[53210]: failed to encode map e82 with expected crc 2024-02-10T06:52:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:36 smithi029 ceph-mon[53210]: failed to encode map e82 with expected crc 2024-02-10T06:52:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:36 smithi029 ceph-mon[53210]: failed to encode map e82 with expected crc 2024-02-10T06:52:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:36 smithi029 ceph-mon[53210]: failed to encode map e82 with expected crc 2024-02-10T06:52:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:36 smithi029 ceph-mon[53210]: failed to encode map e82 with expected crc 2024-02-10T06:52:37.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:36 smithi029 ceph-mon[53210]: failed to encode map e82 with expected crc 2024-02-10T06:52:37.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:36 smithi029 ceph-mon[53210]: osdmap e83: 8 total, 8 up, 8 in 2024-02-10T06:52:37.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 222258 10.70 MB/sec execute 358 sec latency 858.857 ms 2024-02-10T06:52:37.821 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:37 smithi008 ceph-mon[66415]: pgmap v81: 97 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 7 peering, 86 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.0 MiB/s wr, 485 op/s; 87/1500 objects degraded (5.800%); 2.4 MiB/s, 37 objects/s recovering 2024-02-10T06:52:37.821 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:37 smithi008 ceph-mon[66415]: failed to encode map e83 with expected crc 2024-02-10T06:52:37.821 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:37 smithi008 ceph-mon[66415]: failed to encode map e83 with expected crc 2024-02-10T06:52:37.821 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:37 smithi008 ceph-mon[66415]: failed to encode map e83 with expected crc 2024-02-10T06:52:37.821 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:37 smithi008 ceph-mon[66415]: failed to encode map e83 with expected crc 2024-02-10T06:52:37.821 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:37 smithi008 ceph-mon[66415]: osdmap e84: 8 total, 8 up, 8 in 2024-02-10T06:52:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:37 smithi029 ceph-mon[53210]: pgmap v81: 97 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 7 peering, 86 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.0 MiB/s wr, 485 op/s; 87/1500 objects degraded (5.800%); 2.4 MiB/s, 37 objects/s recovering 2024-02-10T06:52:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:37 smithi029 ceph-mon[53210]: failed to encode map e83 with expected crc 2024-02-10T06:52:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:37 smithi029 ceph-mon[53210]: failed to encode map e83 with expected crc 2024-02-10T06:52:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:37 smithi029 ceph-mon[53210]: failed to encode map e83 with expected crc 2024-02-10T06:52:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:37 smithi029 ceph-mon[53210]: failed to encode map e83 with expected crc 2024-02-10T06:52:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:37 smithi029 ceph-mon[53210]: osdmap e84: 8 total, 8 up, 8 in 2024-02-10T06:52:38.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 222580 10.69 MB/sec execute 359 sec latency 757.713 ms 2024-02-10T06:52:38.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: failed to encode map e84 with expected crc 2024-02-10T06:52:38.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: failed to encode map e84 with expected crc 2024-02-10T06:52:38.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: failed to encode map e84 with expected crc 2024-02-10T06:52:38.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: failed to encode map e84 with expected crc 2024-02-10T06:52:38.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: failed to encode map e84 with expected crc 2024-02-10T06:52:38.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:38.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:38.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:52:38.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:52:38.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:38.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:38.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:52:38.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:52:38.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:52:38.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:52:38.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:52:38.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:38 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 87/1500 objects degraded (5.800%), 4 pgs degraded (PG_DEGRADED) 2024-02-10T06:52:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: failed to encode map e84 with expected crc 2024-02-10T06:52:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: failed to encode map e84 with expected crc 2024-02-10T06:52:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: failed to encode map e84 with expected crc 2024-02-10T06:52:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: failed to encode map e84 with expected crc 2024-02-10T06:52:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: failed to encode map e84 with expected crc 2024-02-10T06:52:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:39.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:52:39.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:52:39.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:39.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:39.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:52:39.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:52:39.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:52:39.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:52:39.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:52:39.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:38 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 87/1500 objects degraded (5.800%), 4 pgs degraded (PG_DEGRADED) 2024-02-10T06:52:39.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 222876 10.69 MB/sec execute 360 sec latency 62.380 ms 2024-02-10T06:52:39.926 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:39 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:52:39.926 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:39 smithi008 ceph-mon[66415]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-02-10T06:52:39.926 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:39 smithi008 ceph-mon[66415]: pgmap v84: 97 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 7 peering, 86 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s wr, 3 op/s; 87/1500 objects degraded (5.800%) 2024-02-10T06:52:40.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:39 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:52:40.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:39 smithi029 ceph-mon[53210]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-02-10T06:52:40.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:39 smithi029 ceph-mon[53210]: pgmap v84: 97 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 7 peering, 86 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s wr, 3 op/s; 87/1500 objects degraded (5.800%) 2024-02-10T06:52:40.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 223216 10.70 MB/sec execute 361 sec latency 51.580 ms 2024-02-10T06:52:40.959 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:40 smithi008 ceph-mon[66415]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 87/1500 objects degraded (5.800%), 4 pgs degraded) 2024-02-10T06:52:40.959 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:40 smithi008 ceph-mon[66415]: Cluster is now healthy 2024-02-10T06:52:41.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:40 smithi029 ceph-mon[53210]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 87/1500 objects degraded (5.800%), 4 pgs degraded) 2024-02-10T06:52:41.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:40 smithi029 ceph-mon[53210]: Cluster is now healthy 2024-02-10T06:52:41.661 INFO:teuthology.orchestra.run.smithi008.stdout: 5 223563 10.69 MB/sec execute 362 sec latency 57.230 ms 2024-02-10T06:52:41.982 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:41 smithi008 ceph-mon[66415]: pgmap v85: 97 pgs: 97 active+clean; 350 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.7 MiB/s wr, 387 op/s; 0 B/s, 19 objects/s recovering 2024-02-10T06:52:42.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:41 smithi029 ceph-mon[53210]: pgmap v85: 97 pgs: 97 active+clean; 350 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.7 MiB/s wr, 387 op/s; 0 B/s, 19 objects/s recovering 2024-02-10T06:52:42.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 223956 10.69 MB/sec execute 363 sec latency 38.956 ms 2024-02-10T06:52:43.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 224315 10.69 MB/sec execute 364 sec latency 43.663 ms 2024-02-10T06:52:44.022 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:43 smithi008 ceph-mon[66415]: pgmap v86: 97 pgs: 97 active+clean; 350 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 3.2 MiB/s wr, 326 op/s; 0 B/s, 17 objects/s recovering 2024-02-10T06:52:44.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:43 smithi029 ceph-mon[53210]: pgmap v86: 97 pgs: 97 active+clean; 350 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 3.2 MiB/s wr, 326 op/s; 0 B/s, 17 objects/s recovering 2024-02-10T06:52:44.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 224641 10.68 MB/sec execute 365 sec latency 46.487 ms 2024-02-10T06:52:45.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 224913 10.68 MB/sec execute 366 sec latency 80.037 ms 2024-02-10T06:52:45.847 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:45 smithi008 ceph-mon[66415]: pgmap v87: 97 pgs: 97 active+clean; 353 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.6 MiB/s wr, 563 op/s; 0 B/s, 14 objects/s recovering 2024-02-10T06:52:46.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:45 smithi029 ceph-mon[53210]: pgmap v87: 97 pgs: 97 active+clean; 353 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.6 MiB/s wr, 563 op/s; 0 B/s, 14 objects/s recovering 2024-02-10T06:52:46.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 225119 10.67 MB/sec execute 367 sec latency 96.149 ms 2024-02-10T06:52:47.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 225358 10.65 MB/sec execute 368 sec latency 115.730 ms 2024-02-10T06:52:47.939 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:47 smithi008 ceph-mon[66415]: pgmap v88: 97 pgs: 97 active+clean; 353 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.0 MiB/s wr, 488 op/s; 0 B/s, 11 objects/s recovering 2024-02-10T06:52:48.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:47 smithi029 ceph-mon[53210]: pgmap v88: 97 pgs: 97 active+clean; 353 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.0 MiB/s wr, 488 op/s; 0 B/s, 11 objects/s recovering 2024-02-10T06:52:48.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 225585 10.63 MB/sec execute 369 sec latency 68.721 ms 2024-02-10T06:52:49.459 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:49 smithi008 ceph-mon[66415]: pgmap v89: 97 pgs: 97 active+clean; 353 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.5 MiB/s wr, 442 op/s; 0 B/s, 10 objects/s recovering 2024-02-10T06:52:49.459 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:49.459 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:52:49.459 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:49.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 225922 10.61 MB/sec execute 370 sec latency 41.626 ms 2024-02-10T06:52:49.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:49 smithi029 ceph-mon[53210]: pgmap v89: 97 pgs: 97 active+clean; 353 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.5 MiB/s wr, 442 op/s; 0 B/s, 10 objects/s recovering 2024-02-10T06:52:49.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:49.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:52:49.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:50.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 226354 10.62 MB/sec execute 371 sec latency 44.150 ms 2024-02-10T06:52:51.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:51 smithi008 ceph-mon[66415]: pgmap v90: 97 pgs: 97 active+clean; 348 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.4 MiB/s wr, 551 op/s; 0 B/s, 9 objects/s recovering 2024-02-10T06:52:51.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 226804 10.64 MB/sec execute 372 sec latency 34.514 ms 2024-02-10T06:52:51.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:51 smithi029 ceph-mon[53210]: pgmap v90: 97 pgs: 97 active+clean; 348 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.4 MiB/s wr, 551 op/s; 0 B/s, 9 objects/s recovering 2024-02-10T06:52:52.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 227299 10.64 MB/sec execute 373 sec latency 44.793 ms 2024-02-10T06:52:53.549 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:53 smithi008 ceph-mon[66415]: pgmap v91: 97 pgs: 97 active+clean; 348 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.7 MiB/s wr, 371 op/s 2024-02-10T06:52:53.549 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:53 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:52:53.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 227660 10.64 MB/sec execute 374 sec latency 31.555 ms 2024-02-10T06:52:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:53 smithi029 ceph-mon[53210]: pgmap v91: 97 pgs: 97 active+clean; 348 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.7 MiB/s wr, 371 op/s 2024-02-10T06:52:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:53 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:52:54.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 228039 10.64 MB/sec execute 375 sec latency 39.097 ms 2024-02-10T06:52:54.808 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:54 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:52:54.808 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:54 smithi008 ceph-mon[66415]: Upgrade: osd.1 is safe to restart 2024-02-10T06:52:54.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:54 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T06:52:54.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:54 smithi029 ceph-mon[53210]: Upgrade: osd.1 is safe to restart 2024-02-10T06:52:55.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 228401 10.64 MB/sec execute 376 sec latency 50.446 ms 2024-02-10T06:52:55.819 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:55 smithi008 ceph-mon[66415]: pgmap v92: 97 pgs: 97 active+clean; 356 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.1 MiB/s wr, 619 op/s 2024-02-10T06:52:55.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:55 smithi029 ceph-mon[53210]: pgmap v92: 97 pgs: 97 active+clean; 356 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.1 MiB/s wr, 619 op/s 2024-02-10T06:52:56.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 228665 10.64 MB/sec execute 377 sec latency 74.450 ms 2024-02-10T06:52:56.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:56 smithi008 ceph-mon[66415]: Upgrade: Updating osd.1 2024-02-10T06:52:56.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:56 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:56.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:56 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T06:52:56.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:56 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:52:56.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:56 smithi008 ceph-mon[66415]: Deploying daemon osd.1 on smithi008 2024-02-10T06:52:57.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:56 smithi029 ceph-mon[53210]: Upgrade: Updating osd.1 2024-02-10T06:52:57.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:56 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:52:57.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:56 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T06:52:57.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:56 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:52:57.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:56 smithi029 ceph-mon[53210]: Deploying daemon osd.1 on smithi008 2024-02-10T06:52:57.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 228948 10.62 MB/sec execute 378 sec latency 66.171 ms 2024-02-10T06:52:57.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:57 smithi008 ceph-mon[66415]: pgmap v93: 97 pgs: 97 active+clean; 352 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.4 MiB/s wr, 437 op/s 2024-02-10T06:52:58.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:57 smithi029 ceph-mon[53210]: pgmap v93: 97 pgs: 97 active+clean; 352 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.4 MiB/s wr, 437 op/s 2024-02-10T06:52:58.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 229199 10.61 MB/sec execute 379 sec latency 48.614 ms 2024-02-10T06:52:59.663 INFO:teuthology.orchestra.run.smithi008.stdout: 5 229398 10.59 MB/sec execute 380 sec latency 80.555 ms 2024-02-10T06:52:59.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:52:59 smithi008 ceph-mon[66415]: pgmap v94: 97 pgs: 97 active+clean; 352 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.1 MiB/s wr, 421 op/s 2024-02-10T06:53:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:52:59 smithi029 ceph-mon[53210]: pgmap v94: 97 pgs: 97 active+clean; 352 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.1 MiB/s wr, 421 op/s 2024-02-10T06:53:00.663 INFO:teuthology.orchestra.run.smithi008.stdout: 5 229717 10.58 MB/sec execute 381 sec latency 47.021 ms 2024-02-10T06:53:01.474 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:01 smithi008 systemd[1]: Stopping Ceph osd.1 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:53:01.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 229915 10.57 MB/sec execute 382 sec latency 54.137 ms 2024-02-10T06:53:01.777 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:01 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1[37910]: 2024-02-10T06:53:01.730+0000 7fd0295d6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:53:01.777 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:01 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1[37910]: 2024-02-10T06:53:01.730+0000 7fd0295d6700 -1 osd.1 84 *** Got signal Terminated *** 2024-02-10T06:53:01.777 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:01 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1[37910]: 2024-02-10T06:53:01.730+0000 7fd0295d6700 -1 osd.1 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:53:02.050 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:01 smithi008 ceph-mon[66415]: pgmap v95: 97 pgs: 97 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 5.8 MiB/s wr, 602 op/s 2024-02-10T06:53:02.050 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:01 smithi008 ceph-mon[66415]: osd.1 marked itself down and dead 2024-02-10T06:53:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:01 smithi029 ceph-mon[53210]: pgmap v95: 97 pgs: 97 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 5.8 MiB/s wr, 602 op/s 2024-02-10T06:53:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:01 smithi029 ceph-mon[53210]: osd.1 marked itself down and dead 2024-02-10T06:53:02.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 230058 10.56 MB/sec execute 383 sec latency 818.918 ms 2024-02-10T06:53:02.783 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:02 smithi008 podman[74648]: 2024-02-10 06:53:02.491883928 +0000 UTC m=+0.891672474 container died 0f1119e17a52ef2e3a315be8e1f6932f68d744622157fce57efaaab886c8af30 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1, architecture=x86_64, ceph=True, io.openshift.expose-services=, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, maintainer=Guillaume Abrioux , vcs-type=git, version=8, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-10T06:53:03.080 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:02 smithi008 ceph-mon[66415]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:53:03.080 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:02 smithi008 ceph-mon[66415]: failed to encode map e85 with expected crc 2024-02-10T06:53:03.080 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:02 smithi008 ceph-mon[66415]: osdmap e85: 8 total, 7 up, 8 in 2024-02-10T06:53:03.080 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:02 smithi008 ceph-mon[66415]: failed to encode map e85 with expected crc 2024-02-10T06:53:03.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:02 smithi029 ceph-mon[53210]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:53:03.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:02 smithi029 ceph-mon[53210]: failed to encode map e85 with expected crc 2024-02-10T06:53:03.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:02 smithi029 ceph-mon[53210]: osdmap e85: 8 total, 7 up, 8 in 2024-02-10T06:53:03.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:02 smithi029 ceph-mon[53210]: failed to encode map e85 with expected crc 2024-02-10T06:53:03.585 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:03 smithi008 podman[74648]: 2024-02-10 06:53:03.515566155 +0000 UTC m=+1.915354700 container cleanup 0f1119e17a52ef2e3a315be8e1f6932f68d744622157fce57efaaab886c8af30 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, name=centos-stream, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.openshift.tags=base centos centos-stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, vendor=Red Hat, Inc., release=754, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-10T06:53:03.585 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:03 smithi008 bash[74648]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1 2024-02-10T06:53:03.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 230286 10.54 MB/sec execute 384 sec latency 1014.772 ms 2024-02-10T06:53:03.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:03 smithi008 ceph-mon[66415]: pgmap v97: 97 pgs: 13 stale+active+clean, 84 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.3 MiB/s wr, 530 op/s 2024-02-10T06:53:03.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:03 smithi008 ceph-mon[66415]: failed to encode map e86 with expected crc 2024-02-10T06:53:03.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:03 smithi008 ceph-mon[66415]: osdmap e86: 8 total, 7 up, 8 in 2024-02-10T06:53:03.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:03 smithi008 ceph-mon[66415]: failed to encode map e86 with expected crc 2024-02-10T06:53:03.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:03 smithi008 ceph-mon[66415]: failed to encode map e86 with expected crc 2024-02-10T06:53:03.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:03 smithi008 ceph-mon[66415]: failed to encode map e86 with expected crc 2024-02-10T06:53:03.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:03 smithi008 ceph-mon[66415]: failed to encode map e86 with expected crc 2024-02-10T06:53:03.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:03 smithi008 ceph-mon[66415]: failed to encode map e86 with expected crc 2024-02-10T06:53:03.836 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:03 smithi008 ceph-mon[66415]: failed to encode map e86 with expected crc 2024-02-10T06:53:03.837 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:03 smithi008 ceph-mon[66415]: failed to encode map e86 with expected crc 2024-02-10T06:53:03.837 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:03 smithi008 ceph-mon[66415]: failed to encode map e86 with expected crc 2024-02-10T06:53:03.837 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:03.837 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:53:04.088 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:03 smithi008 podman[74666]: 2024-02-10 06:53:03.913085677 +0000 UTC m=+1.419745312 container remove 0f1119e17a52ef2e3a315be8e1f6932f68d744622157fce57efaaab886c8af30 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, io.buildah.version=1.19.8, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, version=8, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc.) 2024-02-10T06:53:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:03 smithi029 ceph-mon[53210]: pgmap v97: 97 pgs: 13 stale+active+clean, 84 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.3 MiB/s wr, 530 op/s 2024-02-10T06:53:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:03 smithi029 ceph-mon[53210]: failed to encode map e86 with expected crc 2024-02-10T06:53:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:03 smithi029 ceph-mon[53210]: osdmap e86: 8 total, 7 up, 8 in 2024-02-10T06:53:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:03 smithi029 ceph-mon[53210]: failed to encode map e86 with expected crc 2024-02-10T06:53:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:03 smithi029 ceph-mon[53210]: failed to encode map e86 with expected crc 2024-02-10T06:53:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:03 smithi029 ceph-mon[53210]: failed to encode map e86 with expected crc 2024-02-10T06:53:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:03 smithi029 ceph-mon[53210]: failed to encode map e86 with expected crc 2024-02-10T06:53:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:03 smithi029 ceph-mon[53210]: failed to encode map e86 with expected crc 2024-02-10T06:53:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:03 smithi029 ceph-mon[53210]: failed to encode map e86 with expected crc 2024-02-10T06:53:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:03 smithi029 ceph-mon[53210]: failed to encode map e86 with expected crc 2024-02-10T06:53:04.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:03 smithi029 ceph-mon[53210]: failed to encode map e86 with expected crc 2024-02-10T06:53:04.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:04.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:53:04.595 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:04 smithi008 podman[74748]: 2024-02-10 06:53:04.417504891 +0000 UTC m=+0.023047842 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:53:04.595 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:04 smithi008 podman[74748]: 2024-02-10 06:53:04.539066916 +0000 UTC m=+0.144609866 container create 0b64d19ae06117ad7a96123a14fc2250af6b79c9856d4f359339d88f57ed3fb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True) 2024-02-10T06:53:04.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 230670 10.55 MB/sec execute 385 sec latency 49.862 ms 2024-02-10T06:53:05.109 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:04 smithi008 podman[74748]: 2024-02-10 06:53:04.868428933 +0000 UTC m=+0.473971897 container init 0b64d19ae06117ad7a96123a14fc2250af6b79c9856d4f359339d88f57ed3fb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131) 2024-02-10T06:53:05.109 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:04 smithi008 podman[74748]: 2024-02-10 06:53:04.87439163 +0000 UTC m=+0.479934589 container start 0b64d19ae06117ad7a96123a14fc2250af6b79c9856d4f359339d88f57ed3fb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2) 2024-02-10T06:53:05.109 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:05 smithi008 podman[74748]: 2024-02-10 06:53:05.044145529 +0000 UTC m=+0.649688485 container attach 0b64d19ae06117ad7a96123a14fc2250af6b79c9856d4f359339d88f57ed3fb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-02-10T06:53:05.110 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:04 smithi008 ceph-mon[66415]: Health check failed: Degraded data redundancy: 152/1554 objects degraded (9.781%), 21 pgs degraded (PG_DEGRADED) 2024-02-10T06:53:05.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:04 smithi029 ceph-mon[53210]: Health check failed: Degraded data redundancy: 152/1554 objects degraded (9.781%), 21 pgs degraded (PG_DEGRADED) 2024-02-10T06:53:05.371 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:05 smithi008 podman[74748]: 2024-02-10 06:53:05.327729228 +0000 UTC m=+0.933272185 container died 0b64d19ae06117ad7a96123a14fc2250af6b79c9856d4f359339d88f57ed3fb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:53:05.513 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:53:05.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 230947 10.54 MB/sec execute 386 sec latency 40.956 ms 2024-02-10T06:53:06.130 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:05 smithi008 ceph-mon[66415]: pgmap v99: 97 pgs: 21 active+undersized+degraded, 1 stale+active+clean, 14 active+undersized, 61 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.4 MiB/s wr, 475 op/s; 152/1554 objects degraded (9.781%) 2024-02-10T06:53:06.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:05 smithi029 ceph-mon[53210]: pgmap v99: 97 pgs: 21 active+undersized+degraded, 1 stale+active+clean, 14 active+undersized, 61 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.4 MiB/s wr, 475 op/s; 152/1554 objects degraded (9.781%) 2024-02-10T06:53:06.387 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:06 smithi008 podman[74797]: 2024-02-10 06:53:06.284458202 +0000 UTC m=+0.951485328 container remove 0b64d19ae06117ad7a96123a14fc2250af6b79c9856d4f359339d88f57ed3fb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux ) 2024-02-10T06:53:06.387 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:06 smithi008 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-10T06:53:06.387 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:06 smithi008 systemd[1]: Stopped Ceph osd.1 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:53:06.387 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:06 smithi008 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.1.service: Consumed 4min 24.859s CPU time. 2024-02-10T06:53:06.467 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:53:06.467 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (5m) 35s ago 15m 15.5M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:53:06.467 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (87s) 35s ago 15m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T06:53:06.467 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (80s) 75s ago 14m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T06:53:06.467 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (5m) 35s ago 14m 43.8M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:53:06.467 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (9m) 35s ago 8m 38.1M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:53:06.467 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (8m) 75s ago 8m 14.5M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:53:06.467 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (4m) 35s ago 16m 499M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:53:06.468 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (3m) 75s ago 14m 449M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:53:06.468 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (2m) 35s ago 16m 47.3M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:53:06.468 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (2m) 75s ago 14m 35.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:53:06.468 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (8m) 35s ago 8m 269M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:53:06.468 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (8m) 75s ago 8m 84.3M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:53:06.468 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (4m) 35s ago 14m 11.7M - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:53:06.468 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (4m) 75s ago 14m 18.7M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:53:06.468 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (46s) 35s ago 13m 13.7M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 acb0107eae6c 2024-02-10T06:53:06.468 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (13m) 35s ago 13m 373M 1937M 17.2.0 e1d6a67b021e 0f1119e17a52 2024-02-10T06:53:06.468 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (12m) 35s ago 12m 554M 1937M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:53:06.468 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (12m) 35s ago 12m 382M 1937M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:53:06.468 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (11m) 75s ago 11m 607M 2705M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:53:06.468 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (11m) 75s ago 11m 461M 2705M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:53:06.468 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (11m) 75s ago 11m 381M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:53:06.468 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (10m) 75s ago 10m 588M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:53:06.469 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (3m) 35s ago 13m 58.5M - 2.33.4 514e6a882f6e 193819b86dea 2024-02-10T06:53:06.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 231257 10.54 MB/sec execute 387 sec latency 48.478 ms 2024-02-10T06:53:06.895 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:06 smithi008 systemd[1]: Starting Ceph osd.1 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:53:06.895 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:06 smithi008 ceph-mon[66415]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:53:07.069 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:53:07.070 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:53:07.070 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:53:07.070 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:53:07.070 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:53:07.070 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:53:07.070 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:53:07.070 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:53:07.070 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-10T06:53:07.070 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T06:53:07.070 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:53:07.070 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:53:07.070 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:53:07.070 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:53:07.070 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:53:07.070 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-02-10T06:53:07.071 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 5 2024-02-10T06:53:07.071 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:53:07.071 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:53:07.148 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:06 smithi008 podman[74961]: 2024-02-10 06:53:06.844082362 +0000 UTC m=+0.030255225 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:53:07.148 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:07 smithi008 podman[74961]: 2024-02-10 06:53:07.013045401 +0000 UTC m=+0.199218251 container create ec3ea072de4e19a0f8a22160937e53600e5b4f03200f8659adaea80a05a919a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.build-date=20240131, ceph=True, org.label-schema.vendor=CentOS) 2024-02-10T06:53:07.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:06 smithi029 ceph-mon[53210]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:53:07.402 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:07 smithi008 podman[74961]: 2024-02-10 06:53:07.37361145 +0000 UTC m=+0.559784320 container init ec3ea072de4e19a0f8a22160937e53600e5b4f03200f8659adaea80a05a919a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-activate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-290212a, ceph=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-10T06:53:07.402 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:07 smithi008 podman[74961]: 2024-02-10 06:53:07.381896277 +0000 UTC m=+0.568069126 container start ec3ea072de4e19a0f8a22160937e53600e5b4f03200f8659adaea80a05a919a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, ceph=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-10T06:53:07.654 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:07 smithi008 podman[74961]: 2024-02-10 06:53:07.54081472 +0000 UTC m=+0.726987570 container attach ec3ea072de4e19a0f8a22160937e53600e5b4f03200f8659adaea80a05a919a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-activate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, ceph=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-10T06:53:07.662 INFO:teuthology.orchestra.run.smithi008.stdout: 5 231455 10.52 MB/sec execute 388 sec latency 91.614 ms 2024-02-10T06:53:07.669 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:53:07.669 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:53:07.669 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:53:07.669 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:53:07.669 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:53:07.669 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:53:07.669 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-02-10T06:53:07.669 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-02-10T06:53:07.669 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:53:07.669 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "7/23 daemons upgraded", 2024-02-10T06:53:07.669 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:53:07.669 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:53:07.669 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:53:07.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:07 smithi008 ceph-mon[66415]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:53:07.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:07 smithi008 ceph-mon[66415]: pgmap v100: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.7 MiB/s wr, 492 op/s; 174/1584 objects degraded (10.985%) 2024-02-10T06:53:07.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:07 smithi008 ceph-mon[66415]: from='client.34262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:53:07.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:07 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/216664902' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:53:08.176 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:08 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-activate[74980]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T06:53:08.176 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:08 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-activate[74980]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T06:53:08.176 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:08 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-activate[74980]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T06:53:08.176 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:08 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-activate[74980]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T06:53:08.176 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:08 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-activate[74980]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-10T06:53:08.176 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:08 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-activate[74980]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T06:53:08.176 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:08 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-activate[74980]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-10T06:53:08.177 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:08 smithi008 bash[74961]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T06:53:08.177 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:08 smithi008 bash[74961]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T06:53:08.177 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:08 smithi008 bash[74961]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T06:53:08.177 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:08 smithi008 bash[74961]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T06:53:08.177 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:08 smithi008 bash[74961]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-10T06:53:08.177 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:08 smithi008 bash[74961]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T06:53:08.177 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:08 smithi008 bash[74961]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-10T06:53:08.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:07 smithi029 ceph-mon[53210]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:53:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:07 smithi029 ceph-mon[53210]: pgmap v100: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.7 MiB/s wr, 492 op/s; 174/1584 objects degraded (10.985%) 2024-02-10T06:53:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:07 smithi029 ceph-mon[53210]: from='client.34262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:53:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:07 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/216664902' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:53:08.356 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 174/1584 objects degraded (10.985%), 23 pgs degraded 2024-02-10T06:53:08.356 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-10T06:53:08.356 INFO:teuthology.orchestra.run.smithi008.stdout: osd.1 (root=default,host=smithi008) is down 2024-02-10T06:53:08.356 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 174/1584 objects degraded (10.985%), 23 pgs degraded 2024-02-10T06:53:08.356 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.0 is active+undersized+degraded, acting [7,0] 2024-02-10T06:53:08.356 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.2 is active+undersized+degraded, acting [5,6] 2024-02-10T06:53:08.356 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.4 is active+undersized+degraded, acting [0,2] 2024-02-10T06:53:08.356 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.6 is active+undersized+degraded, acting [6,4] 2024-02-10T06:53:08.357 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.8 is active+undersized+degraded, acting [7,0] 2024-02-10T06:53:08.357 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2024-02-10T06:53:08.357 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.a is active+undersized+degraded, acting [3,7] 2024-02-10T06:53:08.357 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.d is active+undersized+degraded, acting [4,3] 2024-02-10T06:53:08.357 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-02-10T06:53:08.357 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.11 is active+undersized+degraded, acting [6,4] 2024-02-10T06:53:08.357 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.15 is active+undersized+degraded, acting [5,0] 2024-02-10T06:53:08.357 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-02-10T06:53:08.357 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-02-10T06:53:08.357 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-02-10T06:53:08.357 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-02-10T06:53:08.357 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.a is active+undersized+degraded, acting [6,4] 2024-02-10T06:53:08.357 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.e is active+undersized+degraded, acting [7,4] 2024-02-10T06:53:08.357 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.16 is active+undersized+degraded, acting [5,7] 2024-02-10T06:53:08.357 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-02-10T06:53:08.357 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-02-10T06:53:08.358 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1a is active+undersized+degraded, acting [4,2] 2024-02-10T06:53:08.358 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1c is active+undersized+degraded, acting [5,4] 2024-02-10T06:53:08.358 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-02-10T06:53:08.431 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:08 smithi008 podman[75143]: 2024-02-10 06:53:08.213308547 +0000 UTC m=+0.028956899 container died ec3ea072de4e19a0f8a22160937e53600e5b4f03200f8659adaea80a05a919a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-activate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-290212a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-10T06:53:08.663 INFO:teuthology.orchestra.run.smithi008.stdout: 5 231667 10.52 MB/sec execute 389 sec latency 81.440 ms 2024-02-10T06:53:08.954 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:08 smithi008 ceph-mon[66415]: from='client.34270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:53:08.954 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:08 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/4272866773' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:53:09.206 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:09 smithi008 podman[75143]: 2024-02-10 06:53:09.150850281 +0000 UTC m=+0.966498625 container remove ec3ea072de4e19a0f8a22160937e53600e5b4f03200f8659adaea80a05a919a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-activate, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131) 2024-02-10T06:53:09.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:08 smithi029 ceph-mon[53210]: from='client.34270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:53:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:08 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/4272866773' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:53:09.457 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:09 smithi008 podman[75199]: 2024-02-10 06:53:09.322286211 +0000 UTC m=+0.030795467 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:53:09.457 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:09 smithi008 podman[75199]: 2024-02-10 06:53:09.450431508 +0000 UTC m=+0.158940753 container create 09363f1c35aece938da7200cdb49d8ff866039e78ec9cc267438a974cad0ae4e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True) 2024-02-10T06:53:09.663 INFO:teuthology.orchestra.run.smithi008.stdout: 5 231905 10.50 MB/sec execute 390 sec latency 50.341 ms 2024-02-10T06:53:09.981 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:09 smithi008 ceph-mon[66415]: pgmap v101: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.2 MiB/s wr, 221 op/s; 174/1584 objects degraded (10.985%) 2024-02-10T06:53:09.981 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:09 smithi008 podman[75199]: 2024-02-10 06:53:09.829983956 +0000 UTC m=+0.538493214 container init 09363f1c35aece938da7200cdb49d8ff866039e78ec9cc267438a974cad0ae4e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, RELEASE=main-290212a, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:53:09.981 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:09 smithi008 podman[75199]: 2024-02-10 06:53:09.835497492 +0000 UTC m=+0.544006739 container start 09363f1c35aece938da7200cdb49d8ff866039e78ec9cc267438a974cad0ae4e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-02-10T06:53:09.981 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:09 smithi008 ceph-osd[75216]: -- 172.21.15.8:0/3199521337 <== mon.1 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5633c64e51e0 con 0x5633c64b7000 2024-02-10T06:53:10.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:09 smithi029 ceph-mon[53210]: pgmap v101: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.2 MiB/s wr, 221 op/s; 174/1584 objects degraded (10.985%) 2024-02-10T06:53:10.246 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:09 smithi008 bash[75199]: 09363f1c35aece938da7200cdb49d8ff866039e78ec9cc267438a974cad0ae4e 2024-02-10T06:53:10.246 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:09 smithi008 systemd[1]: Started Ceph osd.1 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:53:10.497 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:10 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1[75212]: 2024-02-10T06:53:10.426+0000 7f80af74b7c0 -1 Falling back to public interface 2024-02-10T06:53:10.663 INFO:teuthology.orchestra.run.smithi008.stdout: 5 232140 10.50 MB/sec execute 391 sec latency 50.813 ms 2024-02-10T06:53:11.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:11.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:11.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:53:11.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:11 smithi008 ceph-mon[66415]: pgmap v102: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 343 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.6 MiB/s wr, 440 op/s; 172/1560 objects degraded (11.026%) 2024-02-10T06:53:11.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:11.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:11.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:53:11.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:11 smithi029 ceph-mon[53210]: pgmap v102: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 343 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.6 MiB/s wr, 440 op/s; 172/1560 objects degraded (11.026%) 2024-02-10T06:53:11.663 INFO:teuthology.orchestra.run.smithi008.stdout: 5 232319 10.48 MB/sec execute 392 sec latency 94.967 ms 2024-02-10T06:53:12.306 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:12 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 172/1560 objects degraded (11.026%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T06:53:12.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:12 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 172/1560 objects degraded (11.026%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T06:53:12.663 INFO:teuthology.orchestra.run.smithi008.stdout: 5 232789 10.47 MB/sec execute 393 sec latency 54.742 ms 2024-02-10T06:53:13.317 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:13 smithi008 ceph-mon[66415]: pgmap v103: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 343 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.9 MiB/s wr, 372 op/s; 172/1560 objects degraded (11.026%) 2024-02-10T06:53:13.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:13 smithi029 ceph-mon[53210]: pgmap v103: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 343 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.9 MiB/s wr, 372 op/s; 172/1560 objects degraded (11.026%) 2024-02-10T06:53:13.663 INFO:teuthology.orchestra.run.smithi008.stdout: 5 233271 10.48 MB/sec execute 394 sec latency 29.061 ms 2024-02-10T06:53:14.663 INFO:teuthology.orchestra.run.smithi008.stdout: 5 233667 10.48 MB/sec execute 395 sec latency 37.439 ms 2024-02-10T06:53:15.663 INFO:teuthology.orchestra.run.smithi008.stdout: 5 234053 10.49 MB/sec execute 396 sec latency 43.717 ms 2024-02-10T06:53:15.850 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:15 smithi008 ceph-mon[66415]: pgmap v104: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 347 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.2 MiB/s wr, 531 op/s; 181/1629 objects degraded (11.111%) 2024-02-10T06:53:15.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:15 smithi029 ceph-mon[53210]: pgmap v104: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 347 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.2 MiB/s wr, 531 op/s; 181/1629 objects degraded (11.111%) 2024-02-10T06:53:16.662 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:16 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1[75212]: 2024-02-10T06:53:16.592+0000 7f80af74b7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-02-10T06:53:16.663 INFO:teuthology.orchestra.run.smithi008.stdout: 5 234343 10.48 MB/sec execute 397 sec latency 44.077 ms 2024-02-10T06:53:16.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:16 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 183/1620 objects degraded (11.296%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T06:53:17.184 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:16 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 183/1620 objects degraded (11.296%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T06:53:17.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 234584 10.47 MB/sec execute 398 sec latency 75.639 ms 2024-02-10T06:53:17.838 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:17 smithi008 ceph-mon[66415]: pgmap v105: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.0 MiB/s wr, 393 op/s; 183/1620 objects degraded (11.296%) 2024-02-10T06:53:18.194 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:18 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1[75212]: 2024-02-10T06:53:18.156+0000 7f80af74b7c0 -1 osd.1 84 log_to_monitors true 2024-02-10T06:53:18.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:17 smithi029 ceph-mon[53210]: pgmap v105: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.0 MiB/s wr, 393 op/s; 183/1620 objects degraded (11.296%) 2024-02-10T06:53:18.663 INFO:teuthology.orchestra.run.smithi008.stdout: 5 234860 10.46 MB/sec execute 399 sec latency 49.904 ms 2024-02-10T06:53:18.947 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:18 smithi008 ceph-mon[66415]: from='osd.1 [v2:172.21.15.8:6810/4068065199,v1:172.21.15.8:6811/4068065199]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T06:53:18.947 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:18.947 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:53:19.207 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:53:19 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1[75212]: 2024-02-10T06:53:19.046+0000 7f809cb9b700 -1 osd.1 84 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:53:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:18 smithi029 ceph-mon[53210]: from='osd.1 [v2:172.21.15.8:6810/4068065199,v1:172.21.15.8:6811/4068065199]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T06:53:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:53:19.663 INFO:teuthology.orchestra.run.smithi008.stdout: 5 235136 10.45 MB/sec execute 400 sec latency 55.225 ms 2024-02-10T06:53:19.970 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:19 smithi008 ceph-mon[66415]: pgmap v106: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.8 MiB/s wr, 381 op/s; 183/1620 objects degraded (11.296%) 2024-02-10T06:53:19.970 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:19 smithi008 ceph-mon[66415]: from='osd.1 [v2:172.21.15.8:6810/4068065199,v1:172.21.15.8:6811/4068065199]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T06:53:19.970 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:19 smithi008 ceph-mon[66415]: failed to encode map e87 with expected crc 2024-02-10T06:53:19.970 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:19 smithi008 ceph-mon[66415]: osdmap e87: 8 total, 7 up, 8 in 2024-02-10T06:53:19.970 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:19 smithi008 ceph-mon[66415]: from='osd.1 [v2:172.21.15.8:6810/4068065199,v1:172.21.15.8:6811/4068065199]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T06:53:19.970 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:19 smithi008 ceph-mon[66415]: failed to encode map e87 with expected crc 2024-02-10T06:53:19.970 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:19 smithi008 ceph-mon[66415]: from='osd.1 [v2:172.21.15.8:6810/4068065199,v1:172.21.15.8:6811/4068065199]' entity='osd.1' 2024-02-10T06:53:20.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:19 smithi029 ceph-mon[53210]: pgmap v106: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.8 MiB/s wr, 381 op/s; 183/1620 objects degraded (11.296%) 2024-02-10T06:53:20.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:19 smithi029 ceph-mon[53210]: from='osd.1 [v2:172.21.15.8:6810/4068065199,v1:172.21.15.8:6811/4068065199]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T06:53:20.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:19 smithi029 ceph-mon[53210]: failed to encode map e87 with expected crc 2024-02-10T06:53:20.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:19 smithi029 ceph-mon[53210]: osdmap e87: 8 total, 7 up, 8 in 2024-02-10T06:53:20.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:19 smithi029 ceph-mon[53210]: from='osd.1 [v2:172.21.15.8:6810/4068065199,v1:172.21.15.8:6811/4068065199]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T06:53:20.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:19 smithi029 ceph-mon[53210]: failed to encode map e87 with expected crc 2024-02-10T06:53:20.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:19 smithi029 ceph-mon[53210]: from='osd.1 [v2:172.21.15.8:6810/4068065199,v1:172.21.15.8:6811/4068065199]' entity='osd.1' 2024-02-10T06:53:20.663 INFO:teuthology.orchestra.run.smithi008.stdout: 5 235317 10.44 MB/sec execute 401 sec latency 580.762 ms 2024-02-10T06:53:20.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:20 smithi008 ceph-mon[66415]: failed to encode map e87 with expected crc 2024-02-10T06:53:20.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:20 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:20.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:20 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:20.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:20 smithi008 ceph-mon[66415]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:53:20.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:20 smithi008 ceph-mon[66415]: failed to encode map e88 with expected crc 2024-02-10T06:53:20.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:20 smithi008 ceph-mon[66415]: osd.1 [v2:172.21.15.8:6810/4068065199,v1:172.21.15.8:6811/4068065199] boot 2024-02-10T06:53:20.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:20 smithi008 ceph-mon[66415]: osdmap e88: 8 total, 8 up, 8 in 2024-02-10T06:53:20.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:20 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:53:20.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:20 smithi008 ceph-mon[66415]: failed to encode map e88 with expected crc 2024-02-10T06:53:21.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:20 smithi029 ceph-mon[53210]: failed to encode map e87 with expected crc 2024-02-10T06:53:21.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:20 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:21.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:20 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:21.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:20 smithi029 ceph-mon[53210]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:53:21.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:20 smithi029 ceph-mon[53210]: failed to encode map e88 with expected crc 2024-02-10T06:53:21.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:20 smithi029 ceph-mon[53210]: osd.1 [v2:172.21.15.8:6810/4068065199,v1:172.21.15.8:6811/4068065199] boot 2024-02-10T06:53:21.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:20 smithi029 ceph-mon[53210]: osdmap e88: 8 total, 8 up, 8 in 2024-02-10T06:53:21.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:20 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:53:21.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:20 smithi029 ceph-mon[53210]: failed to encode map e88 with expected crc 2024-02-10T06:53:21.663 INFO:teuthology.orchestra.run.smithi008.stdout: 5 235413 10.42 MB/sec execute 402 sec latency 1093.312 ms 2024-02-10T06:53:22.013 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:21 smithi008 ceph-mon[66415]: pgmap v109: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.9 MiB/s wr, 595 op/s; 187/1635 objects degraded (11.437%) 2024-02-10T06:53:22.013 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:21 smithi008 ceph-mon[66415]: osdmap e89: 8 total, 8 up, 8 in 2024-02-10T06:53:22.014 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:21 smithi008 ceph-mon[66415]: failed to encode map e89 with expected crc 2024-02-10T06:53:22.014 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:21 smithi008 ceph-mon[66415]: failed to encode map e89 with expected crc 2024-02-10T06:53:22.014 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:21 smithi008 ceph-mon[66415]: failed to encode map e89 with expected crc 2024-02-10T06:53:22.014 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:21 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:22.014 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:21 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:21 smithi029 ceph-mon[53210]: pgmap v109: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.9 MiB/s wr, 595 op/s; 187/1635 objects degraded (11.437%) 2024-02-10T06:53:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:21 smithi029 ceph-mon[53210]: osdmap e89: 8 total, 8 up, 8 in 2024-02-10T06:53:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:21 smithi029 ceph-mon[53210]: failed to encode map e89 with expected crc 2024-02-10T06:53:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:21 smithi029 ceph-mon[53210]: failed to encode map e89 with expected crc 2024-02-10T06:53:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:21 smithi029 ceph-mon[53210]: failed to encode map e89 with expected crc 2024-02-10T06:53:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:21 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:21 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:22.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 235534 10.40 MB/sec execute 403 sec latency 1612.290 ms 2024-02-10T06:53:22.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:22 smithi008 ceph-mon[66415]: failed to encode map e89 with expected crc 2024-02-10T06:53:22.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:22 smithi008 ceph-mon[66415]: failed to encode map e89 with expected crc 2024-02-10T06:53:22.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:22 smithi008 ceph-mon[66415]: failed to encode map e89 with expected crc 2024-02-10T06:53:22.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:22 smithi008 ceph-mon[66415]: failed to encode map e89 with expected crc 2024-02-10T06:53:22.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:22 smithi008 ceph-mon[66415]: failed to encode map e89 with expected crc 2024-02-10T06:53:22.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:22 smithi008 ceph-mon[66415]: failed to encode map e89 with expected crc 2024-02-10T06:53:22.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:22 smithi008 ceph-mon[66415]: failed to encode map e90 with expected crc 2024-02-10T06:53:22.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:22 smithi008 ceph-mon[66415]: osdmap e90: 8 total, 8 up, 8 in 2024-02-10T06:53:22.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:22 smithi008 ceph-mon[66415]: failed to encode map e90 with expected crc 2024-02-10T06:53:22.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:22 smithi008 ceph-mon[66415]: failed to encode map e90 with expected crc 2024-02-10T06:53:22.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:22 smithi008 ceph-mon[66415]: failed to encode map e90 with expected crc 2024-02-10T06:53:22.779 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:22 smithi008 ceph-mon[66415]: failed to encode map e90 with expected crc 2024-02-10T06:53:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:22 smithi029 ceph-mon[53210]: failed to encode map e89 with expected crc 2024-02-10T06:53:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:22 smithi029 ceph-mon[53210]: failed to encode map e89 with expected crc 2024-02-10T06:53:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:22 smithi029 ceph-mon[53210]: failed to encode map e89 with expected crc 2024-02-10T06:53:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:22 smithi029 ceph-mon[53210]: failed to encode map e89 with expected crc 2024-02-10T06:53:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:22 smithi029 ceph-mon[53210]: failed to encode map e89 with expected crc 2024-02-10T06:53:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:22 smithi029 ceph-mon[53210]: failed to encode map e89 with expected crc 2024-02-10T06:53:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:22 smithi029 ceph-mon[53210]: failed to encode map e90 with expected crc 2024-02-10T06:53:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:22 smithi029 ceph-mon[53210]: osdmap e90: 8 total, 8 up, 8 in 2024-02-10T06:53:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:22 smithi029 ceph-mon[53210]: failed to encode map e90 with expected crc 2024-02-10T06:53:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:22 smithi029 ceph-mon[53210]: failed to encode map e90 with expected crc 2024-02-10T06:53:23.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:22 smithi029 ceph-mon[53210]: failed to encode map e90 with expected crc 2024-02-10T06:53:23.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:22 smithi029 ceph-mon[53210]: failed to encode map e90 with expected crc 2024-02-10T06:53:23.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 235866 10.40 MB/sec execute 404 sec latency 341.993 ms 2024-02-10T06:53:23.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:23 smithi008 ceph-mon[66415]: failed to encode map e90 with expected crc 2024-02-10T06:53:23.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:23 smithi008 ceph-mon[66415]: failed to encode map e90 with expected crc 2024-02-10T06:53:23.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:23 smithi008 ceph-mon[66415]: pgmap v112: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.8 MiB/s wr, 533 op/s; 187/1635 objects degraded (11.437%) 2024-02-10T06:53:23.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:23 smithi008 ceph-mon[66415]: osdmap e91: 8 total, 8 up, 8 in 2024-02-10T06:53:23.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:23 smithi008 ceph-mon[66415]: failed to encode map e91 with expected crc 2024-02-10T06:53:23.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:23 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 187/1635 objects degraded (11.437%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T06:53:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:23 smithi029 ceph-mon[53210]: failed to encode map e90 with expected crc 2024-02-10T06:53:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:23 smithi029 ceph-mon[53210]: failed to encode map e90 with expected crc 2024-02-10T06:53:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:23 smithi029 ceph-mon[53210]: pgmap v112: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.8 MiB/s wr, 533 op/s; 187/1635 objects degraded (11.437%) 2024-02-10T06:53:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:23 smithi029 ceph-mon[53210]: osdmap e91: 8 total, 8 up, 8 in 2024-02-10T06:53:24.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:23 smithi029 ceph-mon[53210]: failed to encode map e91 with expected crc 2024-02-10T06:53:24.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:23 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 187/1635 objects degraded (11.437%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T06:53:24.663 INFO:teuthology.orchestra.run.smithi008.stdout: 5 236134 10.39 MB/sec execute 405 sec latency 759.925 ms 2024-02-10T06:53:24.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:24 smithi008 ceph-mon[66415]: failed to encode map e91 with expected crc 2024-02-10T06:53:24.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:24 smithi008 ceph-mon[66415]: failed to encode map e91 with expected crc 2024-02-10T06:53:24.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:24 smithi008 ceph-mon[66415]: failed to encode map e91 with expected crc 2024-02-10T06:53:24.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:24 smithi008 ceph-mon[66415]: failed to encode map e91 with expected crc 2024-02-10T06:53:24.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:24 smithi008 ceph-mon[66415]: failed to encode map e92 with expected crc 2024-02-10T06:53:24.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:24 smithi008 ceph-mon[66415]: osdmap e92: 8 total, 8 up, 8 in 2024-02-10T06:53:24.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:24 smithi008 ceph-mon[66415]: failed to encode map e92 with expected crc 2024-02-10T06:53:24.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:24 smithi008 ceph-mon[66415]: failed to encode map e92 with expected crc 2024-02-10T06:53:24.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:24 smithi008 ceph-mon[66415]: failed to encode map e92 with expected crc 2024-02-10T06:53:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:24 smithi029 ceph-mon[53210]: failed to encode map e91 with expected crc 2024-02-10T06:53:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:24 smithi029 ceph-mon[53210]: failed to encode map e91 with expected crc 2024-02-10T06:53:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:24 smithi029 ceph-mon[53210]: failed to encode map e91 with expected crc 2024-02-10T06:53:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:24 smithi029 ceph-mon[53210]: failed to encode map e91 with expected crc 2024-02-10T06:53:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:24 smithi029 ceph-mon[53210]: failed to encode map e92 with expected crc 2024-02-10T06:53:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:24 smithi029 ceph-mon[53210]: osdmap e92: 8 total, 8 up, 8 in 2024-02-10T06:53:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:24 smithi029 ceph-mon[53210]: failed to encode map e92 with expected crc 2024-02-10T06:53:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:24 smithi029 ceph-mon[53210]: failed to encode map e92 with expected crc 2024-02-10T06:53:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:24 smithi029 ceph-mon[53210]: failed to encode map e92 with expected crc 2024-02-10T06:53:25.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 236462 10.38 MB/sec execute 406 sec latency 93.059 ms 2024-02-10T06:53:25.884 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:25 smithi008 ceph-mon[66415]: failed to encode map e92 with expected crc 2024-02-10T06:53:25.884 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:25 smithi008 ceph-mon[66415]: failed to encode map e92 with expected crc 2024-02-10T06:53:25.884 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:25 smithi008 ceph-mon[66415]: failed to encode map e92 with expected crc 2024-02-10T06:53:25.884 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:25 smithi008 ceph-mon[66415]: failed to encode map e92 with expected crc 2024-02-10T06:53:25.884 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:25 smithi008 ceph-mon[66415]: failed to encode map e92 with expected crc 2024-02-10T06:53:25.884 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:25 smithi008 ceph-mon[66415]: pgmap v115: 97 pgs: 3 peering, 1 active+undersized+degraded, 2 active+undersized, 91 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.8 MiB/s wr, 363 op/s; 18/1557 objects degraded (1.156%); 22 B/s, 36 objects/s recovering 2024-02-10T06:53:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:25 smithi029 ceph-mon[53210]: failed to encode map e92 with expected crc 2024-02-10T06:53:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:25 smithi029 ceph-mon[53210]: failed to encode map e92 with expected crc 2024-02-10T06:53:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:25 smithi029 ceph-mon[53210]: failed to encode map e92 with expected crc 2024-02-10T06:53:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:25 smithi029 ceph-mon[53210]: failed to encode map e92 with expected crc 2024-02-10T06:53:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:25 smithi029 ceph-mon[53210]: failed to encode map e92 with expected crc 2024-02-10T06:53:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:25 smithi029 ceph-mon[53210]: pgmap v115: 97 pgs: 3 peering, 1 active+undersized+degraded, 2 active+undersized, 91 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.8 MiB/s wr, 363 op/s; 18/1557 objects degraded (1.156%); 22 B/s, 36 objects/s recovering 2024-02-10T06:53:26.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 236873 10.38 MB/sec execute 407 sec latency 51.934 ms 2024-02-10T06:53:26.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:26 smithi008 ceph-mon[66415]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/1557 objects degraded (1.156%), 1 pg degraded) 2024-02-10T06:53:26.889 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:26 smithi008 ceph-mon[66415]: Cluster is now healthy 2024-02-10T06:53:26.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:26 smithi029 ceph-mon[53210]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/1557 objects degraded (1.156%), 1 pg degraded) 2024-02-10T06:53:26.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:26 smithi029 ceph-mon[53210]: Cluster is now healthy 2024-02-10T06:53:27.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 237260 10.38 MB/sec execute 408 sec latency 48.386 ms 2024-02-10T06:53:27.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:27 smithi008 ceph-mon[66415]: pgmap v116: 97 pgs: 3 peering, 94 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.7 MiB/s wr, 304 op/s; 73 KiB/s, 34 objects/s recovering 2024-02-10T06:53:27.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:27.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:27.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:53:27.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:53:27.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:27.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:27.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:53:27.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:53:27.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:53:27.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:53:27.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T06:53:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:27 smithi029 ceph-mon[53210]: pgmap v116: 97 pgs: 3 peering, 94 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.7 MiB/s wr, 304 op/s; 73 KiB/s, 34 objects/s recovering 2024-02-10T06:53:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:53:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:53:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:28.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:53:28.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:53:28.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:53:28.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:53:28.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T06:53:28.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 237615 10.38 MB/sec execute 409 sec latency 40.631 ms 2024-02-10T06:53:28.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:28 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T06:53:28.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:28 smithi008 ceph-mon[66415]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-02-10T06:53:29.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:28 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T06:53:29.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:28 smithi029 ceph-mon[53210]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-02-10T06:53:29.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 238085 10.40 MB/sec execute 410 sec latency 47.317 ms 2024-02-10T06:53:29.937 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:29 smithi008 ceph-mon[66415]: pgmap v117: 97 pgs: 3 peering, 94 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.3 MiB/s wr, 255 op/s; 61 KiB/s, 28 objects/s recovering 2024-02-10T06:53:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:29 smithi029 ceph-mon[53210]: pgmap v117: 97 pgs: 3 peering, 94 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.3 MiB/s wr, 255 op/s; 61 KiB/s, 28 objects/s recovering 2024-02-10T06:53:30.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 238467 10.40 MB/sec execute 411 sec latency 31.976 ms 2024-02-10T06:53:31.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 238673 10.38 MB/sec execute 412 sec latency 69.162 ms 2024-02-10T06:53:31.977 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:31 smithi008 ceph-mon[66415]: pgmap v118: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.2 MiB/s wr, 536 op/s; 1.0 MiB/s, 22 keys/s, 26 objects/s recovering 2024-02-10T06:53:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:31 smithi029 ceph-mon[53210]: pgmap v118: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.2 MiB/s wr, 536 op/s; 1.0 MiB/s, 22 keys/s, 26 objects/s recovering 2024-02-10T06:53:32.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 239003 10.38 MB/sec execute 413 sec latency 44.224 ms 2024-02-10T06:53:33.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 239399 10.37 MB/sec execute 414 sec latency 47.608 ms 2024-02-10T06:53:34.019 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:33 smithi008 ceph-mon[66415]: pgmap v119: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 465 op/s; 928 KiB/s, 19 keys/s, 22 objects/s recovering 2024-02-10T06:53:34.019 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:33 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:34.019 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:33 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:53:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:33 smithi029 ceph-mon[53210]: pgmap v119: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 465 op/s; 928 KiB/s, 19 keys/s, 22 objects/s recovering 2024-02-10T06:53:34.297 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:33 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:34.297 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:33 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:53:34.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 239717 10.37 MB/sec execute 415 sec latency 58.596 ms 2024-02-10T06:53:35.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 240120 10.37 MB/sec execute 416 sec latency 45.263 ms 2024-02-10T06:53:36.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:35 smithi008 ceph-mon[66415]: pgmap v120: 97 pgs: 97 active+clean; 342 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 633 op/s; 838 KiB/s, 17 keys/s, 17 objects/s recovering 2024-02-10T06:53:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:35 smithi029 ceph-mon[53210]: pgmap v120: 97 pgs: 97 active+clean; 342 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 633 op/s; 838 KiB/s, 17 keys/s, 17 objects/s recovering 2024-02-10T06:53:36.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 240515 10.36 MB/sec execute 417 sec latency 43.115 ms 2024-02-10T06:53:37.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 240786 10.35 MB/sec execute 418 sec latency 45.416 ms 2024-02-10T06:53:37.822 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:37 smithi008 ceph-mon[66415]: pgmap v121: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.6 MiB/s wr, 437 op/s; 714 KiB/s, 14 keys/s, 5 objects/s recovering 2024-02-10T06:53:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:37 smithi029 ceph-mon[53210]: pgmap v121: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.6 MiB/s wr, 437 op/s; 714 KiB/s, 14 keys/s, 5 objects/s recovering 2024-02-10T06:53:38.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 241085 10.36 MB/sec execute 419 sec latency 35.865 ms 2024-02-10T06:53:38.887 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:53:39.616 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:39 smithi008 ceph-mon[66415]: pgmap v122: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.4 MiB/s wr, 425 op/s; 683 KiB/s, 14 keys/s, 2 objects/s recovering 2024-02-10T06:53:39.616 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:39 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:39.616 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:39 smithi008 ceph-mon[66415]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:53:39.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 241439 10.37 MB/sec execute 420 sec latency 46.520 ms 2024-02-10T06:53:39.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:39 smithi029 ceph-mon[53210]: pgmap v122: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.4 MiB/s wr, 425 op/s; 683 KiB/s, 14 keys/s, 2 objects/s recovering 2024-02-10T06:53:39.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:39 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:39.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:39 smithi029 ceph-mon[53210]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:53:39.850 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:53:39.850 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (5m) 19s ago 15m 15.6M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:53:39.850 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (2m) 19s ago 15m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T06:53:39.850 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (113s) 108s ago 15m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T06:53:39.850 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (5m) 19s ago 15m 43.1M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:53:39.850 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (9m) 19s ago 9m 38.1M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:53:39.850 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (9m) 108s ago 9m 14.5M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:53:39.850 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (4m) 19s ago 17m 501M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:53:39.850 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (3m) 108s ago 14m 449M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:53:39.850 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (3m) 19s ago 17m 49.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:53:39.851 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (3m) 108s ago 14m 35.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:53:39.851 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (9m) 19s ago 9m 275M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:53:39.851 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (9m) 108s ago 9m 84.3M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:53:39.851 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (5m) 19s ago 15m 10.8M - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:53:39.851 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (4m) 108s ago 14m 18.7M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:53:39.851 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (79s) 19s ago 14m 268M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 acb0107eae6c 2024-02-10T06:53:39.851 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (30s) 19s ago 13m 12.0M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 09363f1c35ae 2024-02-10T06:53:39.851 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (13m) 19s ago 13m 334M 1937M 17.2.0 e1d6a67b021e af09849884bb 2024-02-10T06:53:39.851 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (12m) 19s ago 12m 417M 1937M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:53:39.851 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (12m) 108s ago 12m 607M 2705M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:53:39.851 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (12m) 108s ago 11m 461M 2705M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:53:39.851 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (11m) 108s ago 11m 381M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:53:39.851 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (11m) 108s ago 11m 588M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:53:39.851 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (3m) 19s ago 14m 63.0M - 2.33.4 514e6a882f6e 193819b86dea 2024-02-10T06:53:40.502 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:53:40.502 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:53:40.502 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:53:40.503 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:53:40.503 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:53:40.503 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:53:40.503 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:53:40.503 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:53:40.503 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-10T06:53:40.503 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:53:40.503 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:53:40.503 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:53:40.503 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:53:40.503 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:53:40.503 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:53:40.503 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-02-10T06:53:40.504 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 6 2024-02-10T06:53:40.504 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:53:40.504 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:53:40.642 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:40 smithi008 ceph-mon[66415]: from='client.44195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:53:40.642 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:40 smithi008 ceph-mon[66415]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:53:40.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 241789 10.36 MB/sec execute 421 sec latency 54.522 ms 2024-02-10T06:53:40.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:40 smithi029 ceph-mon[53210]: from='client.44195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:53:40.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:40 smithi029 ceph-mon[53210]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:53:41.045 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:53:41.045 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:53:41.045 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:53:41.045 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:53:41.045 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:53:41.045 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:53:41.046 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-02-10T06:53:41.046 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-02-10T06:53:41.046 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:53:41.046 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "8/23 daemons upgraded", 2024-02-10T06:53:41.046 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:53:41.046 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:53:41.046 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:53:41.649 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:41 smithi008 ceph-mon[66415]: pgmap v123: 97 pgs: 97 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.5 MiB/s wr, 650 op/s; 683 KiB/s, 14 keys/s, 2 objects/s recovering 2024-02-10T06:53:41.649 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:41 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/2247707298' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:53:41.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 242037 10.36 MB/sec execute 422 sec latency 67.756 ms 2024-02-10T06:53:41.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:41 smithi029 ceph-mon[53210]: pgmap v123: 97 pgs: 97 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.5 MiB/s wr, 650 op/s; 683 KiB/s, 14 keys/s, 2 objects/s recovering 2024-02-10T06:53:41.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:41 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/2247707298' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:53:41.746 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:53:42.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:42 smithi008 ceph-mon[66415]: from='client.44205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:53:42.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:42 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/1787797820' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:53:42.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:42 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T06:53:42.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 242275 10.35 MB/sec execute 423 sec latency 52.014 ms 2024-02-10T06:53:42.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:42 smithi029 ceph-mon[53210]: from='client.44205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:53:42.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:42 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/1787797820' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:53:42.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:42 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T06:53:43.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:43 smithi008 ceph-mon[66415]: pgmap v124: 97 pgs: 97 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.2 MiB/s wr, 425 op/s 2024-02-10T06:53:43.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:43 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T06:53:43.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:43 smithi008 ceph-mon[66415]: Upgrade: osd.2 is safe to restart 2024-02-10T06:53:43.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 242519 10.34 MB/sec execute 424 sec latency 61.622 ms 2024-02-10T06:53:43.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:43 smithi029 ceph-mon[53210]: pgmap v124: 97 pgs: 97 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.2 MiB/s wr, 425 op/s 2024-02-10T06:53:43.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:43 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T06:53:43.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:43 smithi029 ceph-mon[53210]: Upgrade: osd.2 is safe to restart 2024-02-10T06:53:44.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 242771 10.32 MB/sec execute 425 sec latency 90.622 ms 2024-02-10T06:53:45.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 243009 10.31 MB/sec execute 426 sec latency 59.565 ms 2024-02-10T06:53:45.831 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:45 smithi008 ceph-mon[66415]: pgmap v125: 97 pgs: 97 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.5 KiB/s rd, 5.8 MiB/s wr, 585 op/s 2024-02-10T06:53:45.831 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:45 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:45.831 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:45 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T06:53:45.831 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:45 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:53:45.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:45 smithi029 ceph-mon[53210]: pgmap v125: 97 pgs: 97 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.5 KiB/s rd, 5.8 MiB/s wr, 585 op/s 2024-02-10T06:53:45.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:45 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:53:45.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:45 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T06:53:45.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:45 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:53:46.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 243206 10.30 MB/sec execute 427 sec latency 79.925 ms 2024-02-10T06:53:46.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:46 smithi008 ceph-mon[66415]: Upgrade: Updating osd.2 2024-02-10T06:53:46.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:46 smithi008 ceph-mon[66415]: Deploying daemon osd.2 on smithi008 2024-02-10T06:53:46.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:46 smithi029 ceph-mon[53210]: Upgrade: Updating osd.2 2024-02-10T06:53:46.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:46 smithi029 ceph-mon[53210]: Deploying daemon osd.2 on smithi008 2024-02-10T06:53:47.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 243392 10.29 MB/sec execute 428 sec latency 84.519 ms 2024-02-10T06:53:47.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:47 smithi008 ceph-mon[66415]: pgmap v126: 97 pgs: 97 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 406 op/s 2024-02-10T06:53:47.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:47 smithi029 ceph-mon[53210]: pgmap v126: 97 pgs: 97 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 406 op/s 2024-02-10T06:53:48.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 243740 10.28 MB/sec execute 429 sec latency 65.433 ms 2024-02-10T06:53:48.684 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:53:48.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:53:49.664 INFO:teuthology.orchestra.run.smithi008.stdout: 5 244098 10.27 MB/sec execute 430 sec latency 49.389 ms 2024-02-10T06:53:49.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:49 smithi008 ceph-mon[66415]: pgmap v127: 97 pgs: 97 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.8 MiB/s wr, 391 op/s 2024-02-10T06:53:49.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:49 smithi029 ceph-mon[53210]: pgmap v127: 97 pgs: 97 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.8 MiB/s wr, 391 op/s 2024-02-10T06:53:50.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 244415 10.27 MB/sec execute 431 sec latency 44.634 ms 2024-02-10T06:53:50.984 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:50 smithi008 systemd[1]: Stopping Ceph osd.2 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:53:51.235 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:51 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2[41772]: 2024-02-10T06:53:51.182+0000 7f4e44c0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:53:51.235 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:51 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2[41772]: 2024-02-10T06:53:51.182+0000 7f4e44c0d700 -1 osd.2 92 *** Got signal Terminated *** 2024-02-10T06:53:51.235 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:51 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2[41772]: 2024-02-10T06:53:51.182+0000 7f4e44c0d700 -1 osd.2 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:53:51.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 244682 10.27 MB/sec execute 432 sec latency 75.410 ms 2024-02-10T06:53:51.740 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:51 smithi008 ceph-mon[66415]: pgmap v128: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.1 MiB/s wr, 542 op/s 2024-02-10T06:53:51.740 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:51 smithi008 ceph-mon[66415]: osd.2 marked itself down and dead 2024-02-10T06:53:51.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:51 smithi029 ceph-mon[53210]: pgmap v128: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.1 MiB/s wr, 542 op/s 2024-02-10T06:53:51.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:51 smithi029 ceph-mon[53210]: osd.2 marked itself down and dead 2024-02-10T06:53:52.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 244770 10.26 MB/sec execute 433 sec latency 875.501 ms 2024-02-10T06:53:52.705 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:52 smithi008 podman[77349]: 2024-02-10 06:53:52.352879447 +0000 UTC m=+1.301103459 container died af09849884bb9117516d04514847b8982ad2aeb61a3ea1c2e4b9cb1834d04a7b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2, ceph=True, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, vcs-type=git, architecture=x86_64, distribution-scope=public, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, vendor=Red Hat, Inc., version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0) 2024-02-10T06:53:52.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:52 smithi029 ceph-mon[53210]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:53:52.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:52 smithi029 ceph-mon[53210]: failed to encode map e93 with expected crc 2024-02-10T06:53:52.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:52 smithi029 ceph-mon[53210]: osdmap e93: 8 total, 7 up, 8 in 2024-02-10T06:53:52.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:52 smithi029 ceph-mon[53210]: failed to encode map e93 with expected crc 2024-02-10T06:53:52.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:52 smithi029 ceph-mon[53210]: failed to encode map e93 with expected crc 2024-02-10T06:53:52.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:52 smithi029 ceph-mon[53210]: failed to encode map e93 with expected crc 2024-02-10T06:53:52.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:52 smithi029 ceph-mon[53210]: failed to encode map e93 with expected crc 2024-02-10T06:53:52.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:52 smithi029 ceph-mon[53210]: failed to encode map e93 with expected crc 2024-02-10T06:53:52.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:52 smithi029 ceph-mon[53210]: failed to encode map e93 with expected crc 2024-02-10T06:53:52.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:52 smithi029 ceph-mon[53210]: osdmap e94: 8 total, 7 up, 8 in 2024-02-10T06:53:52.959 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:52 smithi008 ceph-mon[66415]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:53:52.959 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:52 smithi008 ceph-mon[66415]: failed to encode map e93 with expected crc 2024-02-10T06:53:52.959 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:52 smithi008 ceph-mon[66415]: osdmap e93: 8 total, 7 up, 8 in 2024-02-10T06:53:52.959 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:52 smithi008 ceph-mon[66415]: failed to encode map e93 with expected crc 2024-02-10T06:53:52.959 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:52 smithi008 ceph-mon[66415]: failed to encode map e93 with expected crc 2024-02-10T06:53:52.959 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:52 smithi008 ceph-mon[66415]: failed to encode map e93 with expected crc 2024-02-10T06:53:52.959 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:52 smithi008 ceph-mon[66415]: failed to encode map e93 with expected crc 2024-02-10T06:53:52.960 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:52 smithi008 ceph-mon[66415]: failed to encode map e93 with expected crc 2024-02-10T06:53:52.960 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:52 smithi008 ceph-mon[66415]: failed to encode map e93 with expected crc 2024-02-10T06:53:52.960 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:52 smithi008 ceph-mon[66415]: osdmap e94: 8 total, 7 up, 8 in 2024-02-10T06:53:53.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 245373 10.28 MB/sec execute 434 sec latency 935.398 ms 2024-02-10T06:53:53.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:53 smithi008 ceph-mon[66415]: pgmap v130: 97 pgs: 6 stale+active+clean, 91 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.6 MiB/s wr, 380 op/s 2024-02-10T06:53:53.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:53 smithi008 ceph-mon[66415]: failed to encode map e94 with expected crc 2024-02-10T06:53:53.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:53 smithi008 ceph-mon[66415]: failed to encode map e94 with expected crc 2024-02-10T06:53:53.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:53 smithi008 ceph-mon[66415]: failed to encode map e94 with expected crc 2024-02-10T06:53:53.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:53 smithi008 ceph-mon[66415]: failed to encode map e94 with expected crc 2024-02-10T06:53:53.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:53 smithi008 ceph-mon[66415]: failed to encode map e94 with expected crc 2024-02-10T06:53:53.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:53 smithi008 ceph-mon[66415]: failed to encode map e94 with expected crc 2024-02-10T06:53:53.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:53 smithi008 ceph-mon[66415]: failed to encode map e94 with expected crc 2024-02-10T06:53:53.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:53 smithi008 ceph-mon[66415]: failed to encode map e94 with expected crc 2024-02-10T06:53:53.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:53 smithi008 ceph-mon[66415]: failed to encode map e94 with expected crc 2024-02-10T06:53:53.969 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:53 smithi008 podman[77349]: 2024-02-10 06:53:53.805876824 +0000 UTC m=+2.754100840 container cleanup af09849884bb9117516d04514847b8982ad2aeb61a3ea1c2e4b9cb1834d04a7b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, vcs-type=git, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, RELEASE=HEAD, architecture=x86_64, distribution-scope=public, com.redhat.component=centos-stream-container, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream) 2024-02-10T06:53:53.969 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:53 smithi008 bash[77349]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2 2024-02-10T06:53:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:53 smithi029 ceph-mon[53210]: pgmap v130: 97 pgs: 6 stale+active+clean, 91 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.6 MiB/s wr, 380 op/s 2024-02-10T06:53:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:53 smithi029 ceph-mon[53210]: failed to encode map e94 with expected crc 2024-02-10T06:53:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:53 smithi029 ceph-mon[53210]: failed to encode map e94 with expected crc 2024-02-10T06:53:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:53 smithi029 ceph-mon[53210]: failed to encode map e94 with expected crc 2024-02-10T06:53:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:53 smithi029 ceph-mon[53210]: failed to encode map e94 with expected crc 2024-02-10T06:53:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:53 smithi029 ceph-mon[53210]: failed to encode map e94 with expected crc 2024-02-10T06:53:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:53 smithi029 ceph-mon[53210]: failed to encode map e94 with expected crc 2024-02-10T06:53:54.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:53 smithi029 ceph-mon[53210]: failed to encode map e94 with expected crc 2024-02-10T06:53:54.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:53 smithi029 ceph-mon[53210]: failed to encode map e94 with expected crc 2024-02-10T06:53:54.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:53 smithi029 ceph-mon[53210]: failed to encode map e94 with expected crc 2024-02-10T06:53:54.220 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:54 smithi008 podman[77396]: 2024-02-10 06:53:54.13832886 +0000 UTC m=+1.784554881 container remove af09849884bb9117516d04514847b8982ad2aeb61a3ea1c2e4b9cb1834d04a7b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2, maintainer=Guillaume Abrioux , name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.buildah.version=1.19.8, ceph=True, release=754, build-date=2022-05-03T08:36:31.336870, version=8, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream) 2024-02-10T06:53:54.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 245761 10.28 MB/sec execute 435 sec latency 30.311 ms 2024-02-10T06:53:54.936 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:54 smithi008 ceph-mon[66415]: Health check failed: Degraded data redundancy: 143/1692 objects degraded (8.452%), 19 pgs degraded (PG_DEGRADED) 2024-02-10T06:53:54.936 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:54 smithi008 podman[77475]: 2024-02-10 06:53:54.636217765 +0000 UTC m=+0.021963593 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:53:54.936 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:54 smithi008 podman[77475]: 2024-02-10 06:53:54.766412204 +0000 UTC m=+0.152158011 container create d13e0b351f23290db61a80a10e565185046bd2aecaf6ecf37e2a6721c2def261 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, RELEASE=main-290212a) 2024-02-10T06:53:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:54 smithi029 ceph-mon[53210]: Health check failed: Degraded data redundancy: 143/1692 objects degraded (8.452%), 19 pgs degraded (PG_DEGRADED) 2024-02-10T06:53:55.497 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:55 smithi008 podman[77475]: 2024-02-10 06:53:55.469588292 +0000 UTC m=+0.855334109 container init d13e0b351f23290db61a80a10e565185046bd2aecaf6ecf37e2a6721c2def261 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-10T06:53:55.497 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:55 smithi008 podman[77475]: 2024-02-10 06:53:55.481164493 +0000 UTC m=+0.866910304 container start d13e0b351f23290db61a80a10e565185046bd2aecaf6ecf37e2a6721c2def261 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-290212a) 2024-02-10T06:53:55.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 246157 10.27 MB/sec execute 436 sec latency 53.770 ms 2024-02-10T06:53:55.751 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:55 smithi008 podman[77475]: 2024-02-10 06:53:55.678631212 +0000 UTC m=+1.064377062 container attach d13e0b351f23290db61a80a10e565185046bd2aecaf6ecf37e2a6721c2def261 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131) 2024-02-10T06:53:56.003 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:55 smithi008 podman[77475]: 2024-02-10 06:53:55.94242767 +0000 UTC m=+1.328173480 container died d13e0b351f23290db61a80a10e565185046bd2aecaf6ecf37e2a6721c2def261 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:53:56.004 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:55 smithi008 ceph-mon[66415]: pgmap v132: 97 pgs: 19 active+undersized+degraded, 5 active+undersized, 73 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 527 op/s; 143/1692 objects degraded (8.452%) 2024-02-10T06:53:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:55 smithi029 ceph-mon[53210]: pgmap v132: 97 pgs: 19 active+undersized+degraded, 5 active+undersized, 73 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 527 op/s; 143/1692 objects degraded (8.452%) 2024-02-10T06:53:56.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 246527 10.27 MB/sec execute 437 sec latency 46.978 ms 2024-02-10T06:53:57.025 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:56 smithi008 podman[77500]: 2024-02-10 06:53:56.810692955 +0000 UTC m=+0.853888145 container remove d13e0b351f23290db61a80a10e565185046bd2aecaf6ecf37e2a6721c2def261 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240131, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2) 2024-02-10T06:53:57.025 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:56 smithi008 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-10T06:53:57.025 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:56 smithi008 systemd[1]: Stopped Ceph osd.2 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:53:57.025 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:56 smithi008 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.2.service: Consumed 3min 35.600s CPU time. 2024-02-10T06:53:57.283 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:57 smithi008 systemd[1]: Starting Ceph osd.2 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:53:57.543 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:57 smithi008 podman[77601]: 2024-02-10 06:53:57.318693234 +0000 UTC m=+0.023990732 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:53:57.543 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:57 smithi008 podman[77601]: 2024-02-10 06:53:57.445199325 +0000 UTC m=+0.150496798 container create 47c6cbdbce8c60e963ea34b5d95ca308d3e1ddb517c3c23e4c4f42a3716aa9ea (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-10T06:53:57.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 246772 10.26 MB/sec execute 438 sec latency 54.743 ms 2024-02-10T06:53:57.795 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:57 smithi008 ceph-mon[66415]: pgmap v133: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.5 MiB/s wr, 548 op/s; 155/1683 objects degraded (9.210%) 2024-02-10T06:53:57.796 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:57 smithi008 podman[77601]: 2024-02-10 06:53:57.732337373 +0000 UTC m=+0.437634859 container init 47c6cbdbce8c60e963ea34b5d95ca308d3e1ddb517c3c23e4c4f42a3716aa9ea (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-290212a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, io.buildah.version=1.33.2) 2024-02-10T06:53:57.796 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:57 smithi008 podman[77601]: 2024-02-10 06:53:57.743034342 +0000 UTC m=+0.448331818 container start 47c6cbdbce8c60e963ea34b5d95ca308d3e1ddb517c3c23e4c4f42a3716aa9ea (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:53:58.057 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:57 smithi008 podman[77601]: 2024-02-10 06:53:57.865762449 +0000 UTC m=+0.571059924 container attach 47c6cbdbce8c60e963ea34b5d95ca308d3e1ddb517c3c23e4c4f42a3716aa9ea (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131) 2024-02-10T06:53:58.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:57 smithi029 ceph-mon[53210]: pgmap v133: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.5 MiB/s wr, 548 op/s; 155/1683 objects degraded (9.210%) 2024-02-10T06:53:58.457 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:58 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-activate[77616]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T06:53:58.457 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:58 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-activate[77616]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-10T06:53:58.457 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:58 smithi008 bash[77601]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T06:53:58.457 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:58 smithi008 bash[77601]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-10T06:53:58.457 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:58 smithi008 bash[77601]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-10T06:53:58.457 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:58 smithi008 bash[77601]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T06:53:58.457 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:58 smithi008 bash[77601]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-10T06:53:58.457 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:58 smithi008 bash[77601]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T06:53:58.457 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:58 smithi008 bash[77601]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-10T06:53:58.458 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:58 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-activate[77616]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-10T06:53:58.458 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:58 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-activate[77616]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T06:53:58.458 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:58 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-activate[77616]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-10T06:53:58.458 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:58 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-activate[77616]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T06:53:58.458 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:58 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-activate[77616]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-10T06:53:58.458 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:58 smithi008 podman[77601]: 2024-02-10 06:53:58.369969585 +0000 UTC m=+1.075267058 container died 47c6cbdbce8c60e963ea34b5d95ca308d3e1ddb517c3c23e4c4f42a3716aa9ea (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-activate, ceph=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.schema-version=1.0) 2024-02-10T06:53:58.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 247111 10.26 MB/sec execute 439 sec latency 46.339 ms 2024-02-10T06:53:59.465 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:59 smithi008 podman[77737]: 2024-02-10 06:53:59.372503011 +0000 UTC m=+0.989413956 container remove 47c6cbdbce8c60e963ea34b5d95ca308d3e1ddb517c3c23e4c4f42a3716aa9ea (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-activate, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-10T06:53:59.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 247539 10.25 MB/sec execute 440 sec latency 50.062 ms 2024-02-10T06:53:59.726 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:59 smithi008 podman[77797]: 2024-02-10 06:53:59.537800065 +0000 UTC m=+0.024313434 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:53:59.726 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:59 smithi008 podman[77797]: 2024-02-10 06:53:59.648648306 +0000 UTC m=+0.135161650 container create bd6bbb0220855bf9c1e89cadee5fab7c3a325d0fd768e4ec5b0d8910ab7a1e05 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-10T06:53:59.977 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:53:59 smithi008 ceph-mon[66415]: pgmap v134: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.6 MiB/s wr, 322 op/s; 155/1683 objects degraded (9.210%) 2024-02-10T06:53:59.977 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:59 smithi008 podman[77797]: 2024-02-10 06:53:59.918539176 +0000 UTC m=+0.405052528 container init bd6bbb0220855bf9c1e89cadee5fab7c3a325d0fd768e4ec5b0d8910ab7a1e05 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2) 2024-02-10T06:53:59.977 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:59 smithi008 podman[77797]: 2024-02-10 06:53:59.92297832 +0000 UTC m=+0.409491663 container start bd6bbb0220855bf9c1e89cadee5fab7c3a325d0fd768e4ec5b0d8910ab7a1e05 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True) 2024-02-10T06:53:59.978 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:53:59 smithi008 ceph-osd[77819]: -- 172.21.15.8:0/2395473935 <== mon.0 v2:172.21.15.8:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56367e5e51e0 con 0x56367e5b7000 2024-02-10T06:54:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:53:59 smithi029 ceph-mon[53210]: pgmap v134: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.6 MiB/s wr, 322 op/s; 155/1683 objects degraded (9.210%) 2024-02-10T06:54:00.228 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:54:00 smithi008 bash[77797]: bd6bbb0220855bf9c1e89cadee5fab7c3a325d0fd768e4ec5b0d8910ab7a1e05 2024-02-10T06:54:00.228 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:54:00 smithi008 systemd[1]: Started Ceph osd.2 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:54:00.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 247862 10.25 MB/sec execute 441 sec latency 52.433 ms 2024-02-10T06:54:00.745 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:54:00 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2[77815]: 2024-02-10T06:54:00.502+0000 7f61b951f7c0 -1 Falling back to public interface 2024-02-10T06:54:01.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:01 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:01.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:01 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:01.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:01 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:54:01.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:01 smithi008 ceph-mon[66415]: pgmap v135: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 340 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.1 MiB/s wr, 614 op/s; 151/1596 objects degraded (9.461%) 2024-02-10T06:54:01.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:01 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:01.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:01 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:01.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:01 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:54:01.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:01 smithi029 ceph-mon[53210]: pgmap v135: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 340 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.1 MiB/s wr, 614 op/s; 151/1596 objects degraded (9.461%) 2024-02-10T06:54:01.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 248102 10.24 MB/sec execute 442 sec latency 50.668 ms 2024-02-10T06:54:02.280 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:02 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 151/1596 objects degraded (9.461%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T06:54:02.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:02 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 151/1596 objects degraded (9.461%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T06:54:02.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 248454 10.25 MB/sec execute 443 sec latency 47.993 ms 2024-02-10T06:54:03.304 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:03 smithi008 ceph-mon[66415]: pgmap v136: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 340 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.2 MiB/s wr, 528 op/s; 151/1596 objects degraded (9.461%) 2024-02-10T06:54:03.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:03 smithi029 ceph-mon[53210]: pgmap v136: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 340 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.2 MiB/s wr, 528 op/s; 151/1596 objects degraded (9.461%) 2024-02-10T06:54:03.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 248783 10.25 MB/sec execute 444 sec latency 85.606 ms 2024-02-10T06:54:04.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:04 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:04.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:04 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:54:04.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 249055 10.25 MB/sec execute 445 sec latency 52.600 ms 2024-02-10T06:54:04.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:04 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:04.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:04 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:54:05.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 249215 10.23 MB/sec execute 446 sec latency 48.401 ms 2024-02-10T06:54:05.840 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:05 smithi008 ceph-mon[66415]: pgmap v137: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 345 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.4 MiB/s wr, 632 op/s; 153/1641 objects degraded (9.324%) 2024-02-10T06:54:05.840 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:54:05 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2[77815]: 2024-02-10T06:54:05.760+0000 7f61b951f7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-02-10T06:54:05.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:05 smithi029 ceph-mon[53210]: pgmap v137: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 345 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.4 MiB/s wr, 632 op/s; 153/1641 objects degraded (9.324%) 2024-02-10T06:54:06.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 249529 10.23 MB/sec execute 447 sec latency 35.087 ms 2024-02-10T06:54:06.860 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:06 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 150/1656 objects degraded (9.058%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T06:54:07.113 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:54:06 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2[77815]: 2024-02-10T06:54:06.935+0000 7f61b951f7c0 -1 osd.2 92 log_to_monitors true 2024-02-10T06:54:07.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:06 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 150/1656 objects degraded (9.058%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T06:54:07.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 249835 10.22 MB/sec execute 448 sec latency 83.366 ms 2024-02-10T06:54:07.882 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 06:54:07 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2[77815]: 2024-02-10T06:54:07.780+0000 7f61aa977700 -1 osd.2 92 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:54:07.882 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:07 smithi008 ceph-mon[66415]: pgmap v138: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 345 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.4 MiB/s wr, 423 op/s; 150/1656 objects degraded (9.058%) 2024-02-10T06:54:07.882 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:07 smithi008 ceph-mon[66415]: from='osd.2 [v2:172.21.15.8:6818/4148610164,v1:172.21.15.8:6819/4148610164]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T06:54:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:07 smithi029 ceph-mon[53210]: pgmap v138: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 345 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.4 MiB/s wr, 423 op/s; 150/1656 objects degraded (9.058%) 2024-02-10T06:54:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:07 smithi029 ceph-mon[53210]: from='osd.2 [v2:172.21.15.8:6818/4148610164,v1:172.21.15.8:6819/4148610164]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T06:54:08.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 250138 10.21 MB/sec execute 449 sec latency 59.298 ms 2024-02-10T06:54:08.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:08 smithi008 ceph-mon[66415]: from='osd.2 [v2:172.21.15.8:6818/4148610164,v1:172.21.15.8:6819/4148610164]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T06:54:08.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:08 smithi008 ceph-mon[66415]: osdmap e95: 8 total, 7 up, 8 in 2024-02-10T06:54:08.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:08 smithi008 ceph-mon[66415]: from='osd.2 [v2:172.21.15.8:6818/4148610164,v1:172.21.15.8:6819/4148610164]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T06:54:08.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:08 smithi008 ceph-mon[66415]: failed to encode map e95 with expected crc 2024-02-10T06:54:08.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:08 smithi008 ceph-mon[66415]: failed to encode map e95 with expected crc 2024-02-10T06:54:08.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:08 smithi008 ceph-mon[66415]: failed to encode map e95 with expected crc 2024-02-10T06:54:08.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:08 smithi008 ceph-mon[66415]: failed to encode map e95 with expected crc 2024-02-10T06:54:08.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:08 smithi008 ceph-mon[66415]: failed to encode map e95 with expected crc 2024-02-10T06:54:08.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:08 smithi008 ceph-mon[66415]: failed to encode map e95 with expected crc 2024-02-10T06:54:08.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:08 smithi008 ceph-mon[66415]: failed to encode map e95 with expected crc 2024-02-10T06:54:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:08 smithi029 ceph-mon[53210]: from='osd.2 [v2:172.21.15.8:6818/4148610164,v1:172.21.15.8:6819/4148610164]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T06:54:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:08 smithi029 ceph-mon[53210]: osdmap e95: 8 total, 7 up, 8 in 2024-02-10T06:54:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:08 smithi029 ceph-mon[53210]: from='osd.2 [v2:172.21.15.8:6818/4148610164,v1:172.21.15.8:6819/4148610164]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T06:54:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:08 smithi029 ceph-mon[53210]: failed to encode map e95 with expected crc 2024-02-10T06:54:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:08 smithi029 ceph-mon[53210]: failed to encode map e95 with expected crc 2024-02-10T06:54:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:08 smithi029 ceph-mon[53210]: failed to encode map e95 with expected crc 2024-02-10T06:54:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:08 smithi029 ceph-mon[53210]: failed to encode map e95 with expected crc 2024-02-10T06:54:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:08 smithi029 ceph-mon[53210]: failed to encode map e95 with expected crc 2024-02-10T06:54:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:08 smithi029 ceph-mon[53210]: failed to encode map e95 with expected crc 2024-02-10T06:54:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:08 smithi029 ceph-mon[53210]: failed to encode map e95 with expected crc 2024-02-10T06:54:09.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 250289 10.20 MB/sec execute 450 sec latency 914.663 ms 2024-02-10T06:54:09.923 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: pgmap v140: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.8 MiB/s wr, 490 op/s; 150/1656 objects degraded (9.058%) 2024-02-10T06:54:09.923 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:54:09.923 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: osd.2 [v2:172.21.15.8:6818/4148610164,v1:172.21.15.8:6819/4148610164] boot 2024-02-10T06:54:09.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: osdmap e96: 8 total, 8 up, 8 in 2024-02-10T06:54:09.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:54:09.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: failed to encode map e96 with expected crc 2024-02-10T06:54:09.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: failed to encode map e96 with expected crc 2024-02-10T06:54:09.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: failed to encode map e96 with expected crc 2024-02-10T06:54:09.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: failed to encode map e96 with expected crc 2024-02-10T06:54:09.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: failed to encode map e96 with expected crc 2024-02-10T06:54:09.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: failed to encode map e96 with expected crc 2024-02-10T06:54:09.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: failed to encode map e96 with expected crc 2024-02-10T06:54:09.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: failed to encode map e96 with expected crc 2024-02-10T06:54:09.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: failed to encode map e96 with expected crc 2024-02-10T06:54:09.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:09.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:09.924 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:09 smithi008 ceph-mon[66415]: osdmap e97: 8 total, 8 up, 8 in 2024-02-10T06:54:10.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: pgmap v140: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.8 MiB/s wr, 490 op/s; 150/1656 objects degraded (9.058%) 2024-02-10T06:54:10.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:54:10.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: osd.2 [v2:172.21.15.8:6818/4148610164,v1:172.21.15.8:6819/4148610164] boot 2024-02-10T06:54:10.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: osdmap e96: 8 total, 8 up, 8 in 2024-02-10T06:54:10.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:54:10.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: failed to encode map e96 with expected crc 2024-02-10T06:54:10.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: failed to encode map e96 with expected crc 2024-02-10T06:54:10.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: failed to encode map e96 with expected crc 2024-02-10T06:54:10.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: failed to encode map e96 with expected crc 2024-02-10T06:54:10.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: failed to encode map e96 with expected crc 2024-02-10T06:54:10.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: failed to encode map e96 with expected crc 2024-02-10T06:54:10.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: failed to encode map e96 with expected crc 2024-02-10T06:54:10.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: failed to encode map e96 with expected crc 2024-02-10T06:54:10.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: failed to encode map e96 with expected crc 2024-02-10T06:54:10.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:10.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:10.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:09 smithi029 ceph-mon[53210]: osdmap e97: 8 total, 8 up, 8 in 2024-02-10T06:54:10.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 250363 10.18 MB/sec execute 451 sec latency 1914.688 ms 2024-02-10T06:54:11.014 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:10 smithi008 ceph-mon[66415]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.014 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:10 smithi008 ceph-mon[66415]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.014 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:10 smithi008 ceph-mon[66415]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.015 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:10 smithi008 ceph-mon[66415]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.015 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:10 smithi008 ceph-mon[66415]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.015 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:10 smithi008 ceph-mon[66415]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.015 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:10 smithi008 ceph-mon[66415]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.015 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:10 smithi008 ceph-mon[66415]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.015 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:10 smithi008 ceph-mon[66415]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.015 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:10 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:11.015 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:10 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:11.015 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:10 smithi008 ceph-mon[66415]: osdmap e98: 8 total, 8 up, 8 in 2024-02-10T06:54:11.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:10 smithi029 ceph-mon[53210]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:10 smithi029 ceph-mon[53210]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:10 smithi029 ceph-mon[53210]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:10 smithi029 ceph-mon[53210]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:10 smithi029 ceph-mon[53210]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:10 smithi029 ceph-mon[53210]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:10 smithi029 ceph-mon[53210]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:10 smithi029 ceph-mon[53210]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:10 smithi029 ceph-mon[53210]: failed to encode map e97 with expected crc 2024-02-10T06:54:11.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:10 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:11.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:10 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:11.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:10 smithi029 ceph-mon[53210]: osdmap e98: 8 total, 8 up, 8 in 2024-02-10T06:54:11.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 250577 10.17 MB/sec execute 452 sec latency 2201.630 ms 2024-02-10T06:54:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:11 smithi008 ceph-mon[66415]: pgmap v143: 97 pgs: 1 remapped+peering, 1 activating, 18 peering, 4 active+undersized+degraded, 2 active+undersized, 71 active+clean; 339 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.8 MiB/s wr, 373 op/s; 22/1539 objects degraded (1.429%) 2024-02-10T06:54:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:11 smithi008 ceph-mon[66415]: failed to encode map e98 with expected crc 2024-02-10T06:54:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:11 smithi008 ceph-mon[66415]: failed to encode map e98 with expected crc 2024-02-10T06:54:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:11 smithi008 ceph-mon[66415]: failed to encode map e98 with expected crc 2024-02-10T06:54:11.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:11 smithi008 ceph-mon[66415]: failed to encode map e98 with expected crc 2024-02-10T06:54:11.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:11 smithi008 ceph-mon[66415]: failed to encode map e98 with expected crc 2024-02-10T06:54:11.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:11 smithi008 ceph-mon[66415]: failed to encode map e98 with expected crc 2024-02-10T06:54:11.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:11 smithi008 ceph-mon[66415]: failed to encode map e98 with expected crc 2024-02-10T06:54:12.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:11 smithi029 ceph-mon[53210]: pgmap v143: 97 pgs: 1 remapped+peering, 1 activating, 18 peering, 4 active+undersized+degraded, 2 active+undersized, 71 active+clean; 339 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.8 MiB/s wr, 373 op/s; 22/1539 objects degraded (1.429%) 2024-02-10T06:54:12.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:11 smithi029 ceph-mon[53210]: failed to encode map e98 with expected crc 2024-02-10T06:54:12.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:11 smithi029 ceph-mon[53210]: failed to encode map e98 with expected crc 2024-02-10T06:54:12.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:11 smithi029 ceph-mon[53210]: failed to encode map e98 with expected crc 2024-02-10T06:54:12.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:11 smithi029 ceph-mon[53210]: failed to encode map e98 with expected crc 2024-02-10T06:54:12.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:11 smithi029 ceph-mon[53210]: failed to encode map e98 with expected crc 2024-02-10T06:54:12.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:11 smithi029 ceph-mon[53210]: failed to encode map e98 with expected crc 2024-02-10T06:54:12.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:11 smithi029 ceph-mon[53210]: failed to encode map e98 with expected crc 2024-02-10T06:54:12.243 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:54:12.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 250818 10.16 MB/sec execute 453 sec latency 868.655 ms 2024-02-10T06:54:12.801 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:12 smithi008 ceph-mon[66415]: failed to encode map e99 with expected crc 2024-02-10T06:54:12.801 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:12 smithi008 ceph-mon[66415]: failed to encode map e99 with expected crc 2024-02-10T06:54:12.801 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:12 smithi008 ceph-mon[66415]: failed to encode map e99 with expected crc 2024-02-10T06:54:12.801 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:12 smithi008 ceph-mon[66415]: osdmap e99: 8 total, 8 up, 8 in 2024-02-10T06:54:12.801 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:12 smithi008 ceph-mon[66415]: failed to encode map e99 with expected crc 2024-02-10T06:54:12.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:12 smithi008 ceph-mon[66415]: failed to encode map e99 with expected crc 2024-02-10T06:54:12.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:12 smithi008 ceph-mon[66415]: failed to encode map e99 with expected crc 2024-02-10T06:54:12.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:12 smithi008 ceph-mon[66415]: failed to encode map e99 with expected crc 2024-02-10T06:54:12.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:12 smithi008 ceph-mon[66415]: failed to encode map e99 with expected crc 2024-02-10T06:54:12.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:12 smithi008 ceph-mon[66415]: failed to encode map e99 with expected crc 2024-02-10T06:54:13.149 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:54:13.149 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (6m) 3s ago 16m 15.4M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:54:13.149 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (2m) 3s ago 16m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T06:54:13.149 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (2m) 2m ago 15m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T06:54:13.149 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (6m) 3s ago 15m 42.9M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:54:13.149 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (10m) 3s ago 10m 38.2M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:54:13.149 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (10m) 2m ago 10m 14.5M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:54:13.150 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (5m) 3s ago 17m 502M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:54:13.150 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (4m) 2m ago 15m 449M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:54:13.150 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (4m) 3s ago 17m 50.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:54:13.150 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (3m) 2m ago 15m 35.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:54:13.150 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (9m) 3s ago 9m 267M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:54:13.150 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (9m) 2m ago 9m 84.3M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:54:13.150 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (6m) 3s ago 15m 16.2M - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:54:13.150 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (5m) 2m ago 15m 18.7M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:54:13.150 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (113s) 3s ago 14m 305M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 acb0107eae6c 2024-02-10T06:54:13.150 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (63s) 3s ago 14m 248M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 09363f1c35ae 2024-02-10T06:54:13.150 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (13s) 3s ago 13m 14.9M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 bd6bbb022085 2024-02-10T06:54:13.150 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (13m) 3s ago 13m 469M 1937M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:54:13.151 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (12m) 2m ago 12m 607M 2705M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:54:13.151 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (12m) 2m ago 12m 461M 2705M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:54:13.151 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (12m) 2m ago 12m 381M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:54:13.151 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (11m) 2m ago 11m 588M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:54:13.151 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (4m) 3s ago 15m 63.1M - 2.33.4 514e6a882f6e 193819b86dea 2024-02-10T06:54:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:12 smithi029 ceph-mon[53210]: failed to encode map e99 with expected crc 2024-02-10T06:54:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:12 smithi029 ceph-mon[53210]: failed to encode map e99 with expected crc 2024-02-10T06:54:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:12 smithi029 ceph-mon[53210]: failed to encode map e99 with expected crc 2024-02-10T06:54:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:12 smithi029 ceph-mon[53210]: osdmap e99: 8 total, 8 up, 8 in 2024-02-10T06:54:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:12 smithi029 ceph-mon[53210]: failed to encode map e99 with expected crc 2024-02-10T06:54:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:12 smithi029 ceph-mon[53210]: failed to encode map e99 with expected crc 2024-02-10T06:54:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:12 smithi029 ceph-mon[53210]: failed to encode map e99 with expected crc 2024-02-10T06:54:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:12 smithi029 ceph-mon[53210]: failed to encode map e99 with expected crc 2024-02-10T06:54:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:12 smithi029 ceph-mon[53210]: failed to encode map e99 with expected crc 2024-02-10T06:54:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:12 smithi029 ceph-mon[53210]: failed to encode map e99 with expected crc 2024-02-10T06:54:13.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 251165 10.16 MB/sec execute 454 sec latency 1000.959 ms 2024-02-10T06:54:13.741 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:54:13.741 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:54:13.741 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:54:13.741 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:54:13.741 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:54:13.741 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:54:13.742 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:54:13.742 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:54:13.742 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-02-10T06:54:13.742 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:54:13.742 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:54:13.742 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:54:13.742 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:54:13.742 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:54:13.742 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:54:13.742 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-10T06:54:13.742 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 7 2024-02-10T06:54:13.742 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:54:13.742 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:54:13.810 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:13 smithi008 ceph-mon[66415]: from='client.34308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:54:13.810 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:13 smithi008 ceph-mon[66415]: pgmap v146: 97 pgs: 1 remapped+peering, 1 activating, 18 peering, 4 active+undersized+degraded, 2 active+undersized, 71 active+clean; 339 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.0 MiB/s wr, 526 op/s; 22/1539 objects degraded (1.429%) 2024-02-10T06:54:13.810 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:13 smithi008 ceph-mon[66415]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:54:13.810 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:13 smithi008 ceph-mon[66415]: failed to encode map e100 with expected crc 2024-02-10T06:54:13.810 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:13 smithi008 ceph-mon[66415]: osdmap e100: 8 total, 8 up, 8 in 2024-02-10T06:54:13.810 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:13 smithi008 ceph-mon[66415]: failed to encode map e100 with expected crc 2024-02-10T06:54:13.810 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:13 smithi008 ceph-mon[66415]: failed to encode map e100 with expected crc 2024-02-10T06:54:13.811 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:13 smithi008 ceph-mon[66415]: failed to encode map e100 with expected crc 2024-02-10T06:54:13.811 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:13 smithi008 ceph-mon[66415]: failed to encode map e100 with expected crc 2024-02-10T06:54:13.811 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:13 smithi008 ceph-mon[66415]: failed to encode map e100 with expected crc 2024-02-10T06:54:13.811 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:13 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 22/1539 objects degraded (1.429%), 4 pgs degraded (PG_DEGRADED) 2024-02-10T06:54:13.811 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:13 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/3850504641' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:54:14.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:13 smithi029 ceph-mon[53210]: from='client.34308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:54:14.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:13 smithi029 ceph-mon[53210]: pgmap v146: 97 pgs: 1 remapped+peering, 1 activating, 18 peering, 4 active+undersized+degraded, 2 active+undersized, 71 active+clean; 339 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.0 MiB/s wr, 526 op/s; 22/1539 objects degraded (1.429%) 2024-02-10T06:54:14.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:13 smithi029 ceph-mon[53210]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:54:14.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:13 smithi029 ceph-mon[53210]: failed to encode map e100 with expected crc 2024-02-10T06:54:14.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:13 smithi029 ceph-mon[53210]: osdmap e100: 8 total, 8 up, 8 in 2024-02-10T06:54:14.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:13 smithi029 ceph-mon[53210]: failed to encode map e100 with expected crc 2024-02-10T06:54:14.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:13 smithi029 ceph-mon[53210]: failed to encode map e100 with expected crc 2024-02-10T06:54:14.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:13 smithi029 ceph-mon[53210]: failed to encode map e100 with expected crc 2024-02-10T06:54:14.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:13 smithi029 ceph-mon[53210]: failed to encode map e100 with expected crc 2024-02-10T06:54:14.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:13 smithi029 ceph-mon[53210]: failed to encode map e100 with expected crc 2024-02-10T06:54:14.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:13 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 22/1539 objects degraded (1.429%), 4 pgs degraded (PG_DEGRADED) 2024-02-10T06:54:14.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:13 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/3850504641' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:54:14.324 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:54:14.324 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:54:14.324 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:54:14.324 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:54:14.324 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:54:14.324 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:54:14.324 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-02-10T06:54:14.324 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-02-10T06:54:14.324 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:54:14.324 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "9/23 daemons upgraded", 2024-02-10T06:54:14.325 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:54:14.325 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:54:14.325 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:54:14.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 251475 10.15 MB/sec execute 455 sec latency 51.566 ms 2024-02-10T06:54:14.820 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:14 smithi008 ceph-mon[66415]: failed to encode map e100 with expected crc 2024-02-10T06:54:14.820 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:14 smithi008 ceph-mon[66415]: from='client.44223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:54:14.962 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:54:15.109 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:14 smithi029 ceph-mon[53210]: failed to encode map e100 with expected crc 2024-02-10T06:54:15.145 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:14 smithi029 ceph-mon[53210]: from='client.44223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:54:15.668 INFO:teuthology.orchestra.run.smithi008.stdout: 5 251723 10.13 MB/sec execute 456 sec latency 55.342 ms 2024-02-10T06:54:15.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:15 smithi008 ceph-mon[66415]: pgmap v148: 97 pgs: 1 remapped+peering, 1 activating, 1 peering, 94 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.2 MiB/s wr, 351 op/s; 1.1 MiB/s, 30 objects/s recovering 2024-02-10T06:54:15.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:15 smithi008 ceph-mon[66415]: from='client.34324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:54:15.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:15 smithi008 ceph-mon[66415]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/1539 objects degraded (1.429%), 4 pgs degraded) 2024-02-10T06:54:15.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:15 smithi008 ceph-mon[66415]: Cluster is now healthy 2024-02-10T06:54:15.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:15 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/1149076418' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:54:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:15 smithi029 ceph-mon[53210]: pgmap v148: 97 pgs: 1 remapped+peering, 1 activating, 1 peering, 94 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.2 MiB/s wr, 351 op/s; 1.1 MiB/s, 30 objects/s recovering 2024-02-10T06:54:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:15 smithi029 ceph-mon[53210]: from='client.34324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:54:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:15 smithi029 ceph-mon[53210]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/1539 objects degraded (1.429%), 4 pgs degraded) 2024-02-10T06:54:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:15 smithi029 ceph-mon[53210]: Cluster is now healthy 2024-02-10T06:54:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:15 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/1149076418' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:54:16.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 251993 10.13 MB/sec execute 457 sec latency 50.431 ms 2024-02-10T06:54:17.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 252248 10.13 MB/sec execute 458 sec latency 64.781 ms 2024-02-10T06:54:17.916 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:17 smithi008 ceph-mon[66415]: pgmap v149: 97 pgs: 1 peering, 96 active+clean; 336 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.3 KiB/s rd, 2.3 MiB/s wr, 273 op/s; 903 KiB/s, 25 objects/s recovering 2024-02-10T06:54:17.916 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:17.916 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:17.916 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:54:17.916 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:54:17.916 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:17.917 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:17.917 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:54:17.917 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:54:17.917 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:54:17.917 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:54:17.917 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T06:54:17.917 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:17 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T06:54:17.917 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:17 smithi008 ceph-mon[66415]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-02-10T06:54:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:17 smithi029 ceph-mon[53210]: pgmap v149: 97 pgs: 1 peering, 96 active+clean; 336 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.3 KiB/s rd, 2.3 MiB/s wr, 273 op/s; 903 KiB/s, 25 objects/s recovering 2024-02-10T06:54:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:54:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:54:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:54:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:54:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:54:17.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:54:17.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T06:54:17.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:17 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T06:54:17.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:17 smithi029 ceph-mon[53210]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-02-10T06:54:18.667 INFO:teuthology.orchestra.run.smithi008.stdout: 5 252541 10.14 MB/sec execute 459 sec latency 48.175 ms 2024-02-10T06:54:19.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:19 smithi008 ceph-mon[66415]: pgmap v150: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.3 MiB/s wr, 323 op/s; 1.8 MiB/s, 21 objects/s recovering 2024-02-10T06:54:19.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:19 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:19.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:19 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:54:19.668 INFO:teuthology.orchestra.run.smithi008.stdout: 5 252827 10.13 MB/sec execute 460 sec latency 53.448 ms 2024-02-10T06:54:19.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:19 smithi029 ceph-mon[53210]: pgmap v150: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.3 MiB/s wr, 323 op/s; 1.8 MiB/s, 21 objects/s recovering 2024-02-10T06:54:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:19 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:19 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:54:20.668 INFO:teuthology.orchestra.run.smithi008.stdout: 5 253076 10.13 MB/sec execute 461 sec latency 100.409 ms 2024-02-10T06:54:21.515 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:21 smithi008 ceph-mon[66415]: pgmap v151: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.1 MiB/s wr, 381 op/s; 1.6 MiB/s, 18 objects/s recovering 2024-02-10T06:54:21.668 INFO:teuthology.orchestra.run.smithi008.stdout: 5 253276 10.12 MB/sec execute 462 sec latency 50.309 ms 2024-02-10T06:54:21.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:21 smithi029 ceph-mon[53210]: pgmap v151: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.1 MiB/s wr, 381 op/s; 1.6 MiB/s, 18 objects/s recovering 2024-02-10T06:54:22.668 INFO:teuthology.orchestra.run.smithi008.stdout: 5 253495 10.10 MB/sec execute 463 sec latency 58.979 ms 2024-02-10T06:54:23.543 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:23 smithi008 ceph-mon[66415]: pgmap v152: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.5 MiB/s wr, 326 op/s; 1.3 MiB/s, 15 objects/s recovering 2024-02-10T06:54:23.668 INFO:teuthology.orchestra.run.smithi008.stdout: 5 253823 10.10 MB/sec execute 464 sec latency 46.866 ms 2024-02-10T06:54:23.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:23 smithi029 ceph-mon[53210]: pgmap v152: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.5 MiB/s wr, 326 op/s; 1.3 MiB/s, 15 objects/s recovering 2024-02-10T06:54:24.551 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:24 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:24.668 INFO:teuthology.orchestra.run.smithi008.stdout: 5 254297 10.11 MB/sec execute 465 sec latency 42.812 ms 2024-02-10T06:54:24.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:24 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:25.668 INFO:teuthology.orchestra.run.smithi008.stdout: 5 254833 10.11 MB/sec execute 466 sec latency 78.795 ms 2024-02-10T06:54:25.813 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:25 smithi008 ceph-mon[66415]: pgmap v153: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.8 MiB/s wr, 484 op/s; 1.2 MiB/s, 13 objects/s recovering 2024-02-10T06:54:25.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:25 smithi029 ceph-mon[53210]: pgmap v153: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.8 MiB/s wr, 484 op/s; 1.2 MiB/s, 13 objects/s recovering 2024-02-10T06:54:26.668 INFO:teuthology.orchestra.run.smithi008.stdout: 5 255226 10.10 MB/sec execute 467 sec latency 45.699 ms 2024-02-10T06:54:27.668 INFO:teuthology.orchestra.run.smithi008.stdout: 5 255540 10.10 MB/sec execute 468 sec latency 78.472 ms 2024-02-10T06:54:27.891 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:27 smithi008 ceph-mon[66415]: pgmap v154: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.9 MiB/s wr, 354 op/s; 687 KiB/s, 1 objects/s recovering 2024-02-10T06:54:27.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:27 smithi029 ceph-mon[53210]: pgmap v154: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.9 MiB/s wr, 354 op/s; 687 KiB/s, 1 objects/s recovering 2024-02-10T06:54:28.668 INFO:teuthology.orchestra.run.smithi008.stdout: 5 255932 10.11 MB/sec execute 469 sec latency 44.756 ms 2024-02-10T06:54:29.668 INFO:teuthology.orchestra.run.smithi008.stdout: 5 256271 10.11 MB/sec execute 470 sec latency 44.524 ms 2024-02-10T06:54:29.928 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:29 smithi008 ceph-mon[66415]: pgmap v155: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.7 MiB/s wr, 347 op/s; 682 KiB/s, 0 objects/s recovering 2024-02-10T06:54:29.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:29 smithi029 ceph-mon[53210]: pgmap v155: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.7 MiB/s wr, 347 op/s; 682 KiB/s, 0 objects/s recovering 2024-02-10T06:54:30.668 INFO:teuthology.orchestra.run.smithi008.stdout: 5 256553 10.12 MB/sec execute 471 sec latency 51.078 ms 2024-02-10T06:54:31.668 INFO:teuthology.orchestra.run.smithi008.stdout: 5 256798 10.11 MB/sec execute 472 sec latency 65.144 ms 2024-02-10T06:54:31.751 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:31 smithi008 ceph-mon[66415]: pgmap v156: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.8 MiB/s wr, 509 op/s 2024-02-10T06:54:31.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:31 smithi029 ceph-mon[53210]: pgmap v156: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.8 MiB/s wr, 509 op/s 2024-02-10T06:54:32.668 INFO:teuthology.orchestra.run.smithi008.stdout: 5 257030 10.09 MB/sec execute 473 sec latency 47.601 ms 2024-02-10T06:54:32.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T06:54:32.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:32 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T06:54:32.778 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:32 smithi008 ceph-mon[66415]: Upgrade: osd.3 is safe to restart 2024-02-10T06:54:32.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T06:54:32.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:32 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T06:54:32.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:32 smithi029 ceph-mon[53210]: Upgrade: osd.3 is safe to restart 2024-02-10T06:54:33.669 INFO:teuthology.orchestra.run.smithi008.stdout: 5 257269 10.07 MB/sec execute 474 sec latency 62.849 ms 2024-02-10T06:54:33.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:33 smithi008 ceph-mon[66415]: pgmap v157: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.3 MiB/s wr, 441 op/s 2024-02-10T06:54:33.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:33 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:54:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:33 smithi029 ceph-mon[53210]: pgmap v157: 97 pgs: 97 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.3 MiB/s wr, 441 op/s 2024-02-10T06:54:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:33 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:54:34.669 INFO:teuthology.orchestra.run.smithi008.stdout: 5 257496 10.07 MB/sec execute 475 sec latency 67.079 ms 2024-02-10T06:54:35.669 INFO:teuthology.orchestra.run.smithi008.stdout: 5 257780 10.07 MB/sec execute 476 sec latency 42.428 ms 2024-02-10T06:54:35.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:35 smithi008 ceph-mon[66415]: pgmap v158: 97 pgs: 97 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.8 MiB/s wr, 613 op/s 2024-02-10T06:54:35.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:35 smithi008 ceph-mon[66415]: Upgrade: Updating osd.3 2024-02-10T06:54:35.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:35 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:35.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:35 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T06:54:35.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:35 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:54:35.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:35 smithi008 ceph-mon[66415]: Deploying daemon osd.3 on smithi008 2024-02-10T06:54:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:35 smithi029 ceph-mon[53210]: pgmap v158: 97 pgs: 97 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.8 MiB/s wr, 613 op/s 2024-02-10T06:54:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:35 smithi029 ceph-mon[53210]: Upgrade: Updating osd.3 2024-02-10T06:54:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:35 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:35 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T06:54:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:35 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:54:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:35 smithi029 ceph-mon[53210]: Deploying daemon osd.3 on smithi008 2024-02-10T06:54:36.669 INFO:teuthology.orchestra.run.smithi008.stdout: 5 258119 10.06 MB/sec execute 477 sec latency 103.816 ms 2024-02-10T06:54:37.669 INFO:teuthology.orchestra.run.smithi008.stdout: 5 258580 10.06 MB/sec execute 478 sec latency 48.766 ms 2024-02-10T06:54:38.022 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:37 smithi008 ceph-mon[66415]: pgmap v159: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 429 op/s 2024-02-10T06:54:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:37 smithi029 ceph-mon[53210]: pgmap v159: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 429 op/s 2024-02-10T06:54:38.669 INFO:teuthology.orchestra.run.smithi008.stdout: 5 258937 10.05 MB/sec execute 479 sec latency 28.723 ms 2024-02-10T06:54:39.669 INFO:teuthology.orchestra.run.smithi008.stdout: 5 259320 10.06 MB/sec execute 480 sec latency 44.855 ms 2024-02-10T06:54:40.055 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:39 smithi008 ceph-mon[66415]: pgmap v160: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.0 MiB/s wr, 416 op/s 2024-02-10T06:54:40.056 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:39 smithi008 systemd[1]: Stopping Ceph osd.3 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:54:40.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:39 smithi029 ceph-mon[53210]: pgmap v160: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.0 MiB/s wr, 416 op/s 2024-02-10T06:54:40.306 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:40 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3[45519]: 2024-02-10T06:54:40.280+0000 7f01f7e19700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:54:40.306 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:40 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3[45519]: 2024-02-10T06:54:40.280+0000 7f01f7e19700 -1 osd.3 100 *** Got signal Terminated *** 2024-02-10T06:54:40.306 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:40 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3[45519]: 2024-02-10T06:54:40.280+0000 7f01f7e19700 -1 osd.3 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:54:40.669 INFO:teuthology.orchestra.run.smithi008.stdout: 5 259859 10.08 MB/sec execute 481 sec latency 40.378 ms 2024-02-10T06:54:41.066 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:40 smithi008 ceph-mon[66415]: osd.3 marked itself down and dead 2024-02-10T06:54:41.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:40 smithi029 ceph-mon[53210]: osd.3 marked itself down and dead 2024-02-10T06:54:41.669 INFO:teuthology.orchestra.run.smithi008.stdout: 5 260006 10.08 MB/sec execute 482 sec latency 750.278 ms 2024-02-10T06:54:41.754 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:41 smithi008 podman[79958]: 2024-02-10 06:54:41.536532408 +0000 UTC m=+1.415224990 container died 1801788238d620f1084b565084c0340d5c9110e1668b4040c0ee8bfd1bb530fe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, architecture=x86_64, distribution-scope=public, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.openshift.expose-services=, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, version=8, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-10T06:54:42.004 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:41 smithi008 ceph-mon[66415]: pgmap v161: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.1 MiB/s wr, 640 op/s 2024-02-10T06:54:42.004 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:41 smithi008 ceph-mon[66415]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:54:42.004 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:41 smithi008 ceph-mon[66415]: osdmap e101: 8 total, 7 up, 8 in 2024-02-10T06:54:42.004 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:41 smithi008 ceph-mon[66415]: failed to encode map e101 with expected crc 2024-02-10T06:54:42.004 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:41 smithi008 ceph-mon[66415]: failed to encode map e101 with expected crc 2024-02-10T06:54:42.004 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:41 smithi008 ceph-mon[66415]: failed to encode map e101 with expected crc 2024-02-10T06:54:42.005 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:41 smithi008 ceph-mon[66415]: failed to encode map e101 with expected crc 2024-02-10T06:54:42.005 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:41 smithi008 ceph-mon[66415]: failed to encode map e101 with expected crc 2024-02-10T06:54:42.005 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:41 smithi008 ceph-mon[66415]: failed to encode map e101 with expected crc 2024-02-10T06:54:42.005 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:41 smithi008 podman[79958]: 2024-02-10 06:54:41.994821938 +0000 UTC m=+1.873514508 container cleanup 1801788238d620f1084b565084c0340d5c9110e1668b4040c0ee8bfd1bb530fe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, maintainer=Guillaume Abrioux , version=8, io.openshift.expose-services=, GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, release=754, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, name=centos-stream, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc.) 2024-02-10T06:54:42.005 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:41 smithi008 bash[79958]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3 2024-02-10T06:54:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:41 smithi029 ceph-mon[53210]: pgmap v161: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.1 MiB/s wr, 640 op/s 2024-02-10T06:54:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:41 smithi029 ceph-mon[53210]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:54:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:41 smithi029 ceph-mon[53210]: osdmap e101: 8 total, 7 up, 8 in 2024-02-10T06:54:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:41 smithi029 ceph-mon[53210]: failed to encode map e101 with expected crc 2024-02-10T06:54:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:41 smithi029 ceph-mon[53210]: failed to encode map e101 with expected crc 2024-02-10T06:54:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:41 smithi029 ceph-mon[53210]: failed to encode map e101 with expected crc 2024-02-10T06:54:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:41 smithi029 ceph-mon[53210]: failed to encode map e101 with expected crc 2024-02-10T06:54:42.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:41 smithi029 ceph-mon[53210]: failed to encode map e101 with expected crc 2024-02-10T06:54:42.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:41 smithi029 ceph-mon[53210]: failed to encode map e101 with expected crc 2024-02-10T06:54:42.507 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:42 smithi008 podman[79973]: 2024-02-10 06:54:42.305205394 +0000 UTC m=+0.773177486 container remove 1801788238d620f1084b565084c0340d5c9110e1668b4040c0ee8bfd1bb530fe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, name=centos-stream, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.buildah.version=1.19.8, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., architecture=x86_64, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux ) 2024-02-10T06:54:42.669 INFO:teuthology.orchestra.run.smithi008.stdout: 5 260376 10.07 MB/sec execute 483 sec latency 967.676 ms 2024-02-10T06:54:42.760 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:42 smithi008 podman[80060]: 2024-02-10 06:54:42.654273478 +0000 UTC m=+0.027255879 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:54:43.011 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:42 smithi008 podman[80060]: 2024-02-10 06:54:42.804042212 +0000 UTC m=+0.177024630 container create a2d12fd3f9760ea21d48ad22f156469561f3542636094844d5a53589b25ae3a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=) 2024-02-10T06:54:43.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:42 smithi008 ceph-mon[66415]: failed to encode map e102 with expected crc 2024-02-10T06:54:43.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:42 smithi008 ceph-mon[66415]: osdmap e102: 8 total, 7 up, 8 in 2024-02-10T06:54:43.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:42 smithi008 ceph-mon[66415]: failed to encode map e102 with expected crc 2024-02-10T06:54:43.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:42 smithi008 ceph-mon[66415]: failed to encode map e102 with expected crc 2024-02-10T06:54:43.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:42 smithi008 ceph-mon[66415]: failed to encode map e102 with expected crc 2024-02-10T06:54:43.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:42 smithi008 ceph-mon[66415]: failed to encode map e102 with expected crc 2024-02-10T06:54:43.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:42 smithi008 ceph-mon[66415]: failed to encode map e102 with expected crc 2024-02-10T06:54:43.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:42 smithi008 ceph-mon[66415]: failed to encode map e102 with expected crc 2024-02-10T06:54:43.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:42 smithi008 ceph-mon[66415]: failed to encode map e102 with expected crc 2024-02-10T06:54:43.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:42 smithi029 ceph-mon[53210]: failed to encode map e102 with expected crc 2024-02-10T06:54:43.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:42 smithi029 ceph-mon[53210]: osdmap e102: 8 total, 7 up, 8 in 2024-02-10T06:54:43.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:42 smithi029 ceph-mon[53210]: failed to encode map e102 with expected crc 2024-02-10T06:54:43.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:42 smithi029 ceph-mon[53210]: failed to encode map e102 with expected crc 2024-02-10T06:54:43.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:42 smithi029 ceph-mon[53210]: failed to encode map e102 with expected crc 2024-02-10T06:54:43.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:42 smithi029 ceph-mon[53210]: failed to encode map e102 with expected crc 2024-02-10T06:54:43.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:42 smithi029 ceph-mon[53210]: failed to encode map e102 with expected crc 2024-02-10T06:54:43.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:42 smithi029 ceph-mon[53210]: failed to encode map e102 with expected crc 2024-02-10T06:54:43.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:42 smithi029 ceph-mon[53210]: failed to encode map e102 with expected crc 2024-02-10T06:54:43.265 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:43 smithi008 podman[80060]: 2024-02-10 06:54:43.202631649 +0000 UTC m=+0.575614030 container init a2d12fd3f9760ea21d48ad22f156469561f3542636094844d5a53589b25ae3a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-290212a, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:54:43.265 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:43 smithi008 podman[80060]: 2024-02-10 06:54:43.210279818 +0000 UTC m=+0.583262187 container start a2d12fd3f9760ea21d48ad22f156469561f3542636094844d5a53589b25ae3a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-10T06:54:43.518 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:43 smithi008 podman[80060]: 2024-02-10 06:54:43.328480065 +0000 UTC m=+0.701462436 container attach a2d12fd3f9760ea21d48ad22f156469561f3542636094844d5a53589b25ae3a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240131, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, RELEASE=main-290212a) 2024-02-10T06:54:43.669 INFO:teuthology.orchestra.run.smithi008.stdout: 5 260645 10.06 MB/sec execute 484 sec latency 36.554 ms 2024-02-10T06:54:43.745 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:43 smithi008 podman[80060]: 2024-02-10 06:54:43.637553486 +0000 UTC m=+1.010535872 container died a2d12fd3f9760ea21d48ad22f156469561f3542636094844d5a53589b25ae3a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a) 2024-02-10T06:54:44.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:43 smithi008 ceph-mon[66415]: pgmap v164: 97 pgs: 16 stale+active+clean, 81 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.4 MiB/s wr, 359 op/s 2024-02-10T06:54:44.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:43 smithi029 ceph-mon[53210]: pgmap v164: 97 pgs: 16 stale+active+clean, 81 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.4 MiB/s wr, 359 op/s 2024-02-10T06:54:44.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 260864 10.05 MB/sec execute 485 sec latency 47.805 ms 2024-02-10T06:54:44.824 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:44 smithi008 podman[80085]: 2024-02-10 06:54:44.560644606 +0000 UTC m=+0.916898075 container remove a2d12fd3f9760ea21d48ad22f156469561f3542636094844d5a53589b25ae3a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True) 2024-02-10T06:54:44.824 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:44 smithi008 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.3.service: Deactivated successfully. 2024-02-10T06:54:44.824 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:44 smithi008 systemd[1]: Stopped Ceph osd.3 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:54:44.824 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:44 smithi008 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.3.service: Consumed 5min 31.561s CPU time. 2024-02-10T06:54:45.075 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:44 smithi008 systemd[1]: Starting Ceph osd.3 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:54:45.075 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:44 smithi008 ceph-mon[66415]: Health check failed: Degraded data redundancy: 139/1710 objects degraded (8.129%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T06:54:45.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:44 smithi029 ceph-mon[53210]: Health check failed: Degraded data redundancy: 139/1710 objects degraded (8.129%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T06:54:45.333 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:45 smithi008 podman[80182]: 2024-02-10 06:54:45.05018502 +0000 UTC m=+0.027202924 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:54:45.333 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:45 smithi008 podman[80182]: 2024-02-10 06:54:45.270479609 +0000 UTC m=+0.247497496 container create 164f0c68d0cd73a496dea05b0d4f23cf4215a18b54e05569a219bc17b2585353 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:54:45.432 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:54:45.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 261177 10.05 MB/sec execute 486 sec latency 81.574 ms 2024-02-10T06:54:45.840 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:45 smithi008 podman[80182]: 2024-02-10 06:54:45.61827402 +0000 UTC m=+0.595291915 container init 164f0c68d0cd73a496dea05b0d4f23cf4215a18b54e05569a219bc17b2585353 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-290212a, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True) 2024-02-10T06:54:45.840 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:45 smithi008 podman[80182]: 2024-02-10 06:54:45.626123919 +0000 UTC m=+0.603141800 container start 164f0c68d0cd73a496dea05b0d4f23cf4215a18b54e05569a219bc17b2585353 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-activate, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-10T06:54:45.840 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:45 smithi008 podman[80182]: 2024-02-10 06:54:45.791388344 +0000 UTC m=+0.768406233 container attach 164f0c68d0cd73a496dea05b0d4f23cf4215a18b54e05569a219bc17b2585353 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-activate, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240131) 2024-02-10T06:54:46.100 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:45 smithi008 ceph-mon[66415]: pgmap v165: 97 pgs: 20 active+undersized+degraded, 3 stale+active+clean, 12 active+undersized, 62 active+clean; 347 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.9 MiB/s wr, 596 op/s; 139/1710 objects degraded (8.129%) 2024-02-10T06:54:46.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:45 smithi029 ceph-mon[53210]: pgmap v165: 97 pgs: 20 active+undersized+degraded, 3 stale+active+clean, 12 active+undersized, 62 active+clean; 347 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.9 MiB/s wr, 596 op/s; 139/1710 objects degraded (8.129%) 2024-02-10T06:54:46.352 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:46 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-activate[80219]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T06:54:46.352 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:46 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-activate[80219]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T06:54:46.352 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:46 smithi008 bash[80182]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T06:54:46.352 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:46 smithi008 bash[80182]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T06:54:46.352 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:46 smithi008 bash[80182]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T06:54:46.352 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:46 smithi008 bash[80182]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T06:54:46.352 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:46 smithi008 bash[80182]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-10T06:54:46.352 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:46 smithi008 bash[80182]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T06:54:46.352 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:46 smithi008 bash[80182]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-10T06:54:46.352 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:46 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-activate[80219]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T06:54:46.352 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:46 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-activate[80219]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T06:54:46.352 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:46 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-activate[80219]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-10T06:54:46.353 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:46 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-activate[80219]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T06:54:46.353 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:46 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-activate[80219]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-10T06:54:46.353 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:46 smithi008 podman[80182]: 2024-02-10 06:54:46.268086467 +0000 UTC m=+1.245104351 container died 164f0c68d0cd73a496dea05b0d4f23cf4215a18b54e05569a219bc17b2585353 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2) 2024-02-10T06:54:46.475 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:54:46.475 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (7m) 37s ago 16m 15.4M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:54:46.475 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (3m) 37s ago 16m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T06:54:46.476 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (3m) 2m ago 16m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T06:54:46.476 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (6m) 37s ago 16m 42.9M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:54:46.476 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (10m) 37s ago 10m 38.2M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:54:46.476 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (10m) 2m ago 10m 14.5M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:54:46.476 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (5m) 37s ago 18m 502M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:54:46.476 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (4m) 2m ago 16m 449M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:54:46.476 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (4m) 37s ago 18m 50.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:54:46.476 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (4m) 2m ago 15m 35.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:54:46.476 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (10m) 37s ago 10m 267M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:54:46.476 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (10m) 2m ago 10m 84.3M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:54:46.476 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (6m) 37s ago 16m 16.2M - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:54:46.476 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (6m) 2m ago 15m 18.7M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:54:46.476 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (2m) 37s ago 15m 305M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 acb0107eae6c 2024-02-10T06:54:46.476 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (97s) 37s ago 14m 248M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 09363f1c35ae 2024-02-10T06:54:46.477 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (46s) 37s ago 14m 14.9M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 bd6bbb022085 2024-02-10T06:54:46.477 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (13m) 37s ago 13m 469M 1937M 17.2.0 e1d6a67b021e 1801788238d6 2024-02-10T06:54:46.477 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (13m) 2m ago 13m 607M 2705M 17.2.0 e1d6a67b021e a32c1bdd2400 2024-02-10T06:54:46.477 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (13m) 2m ago 13m 461M 2705M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:54:46.477 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (12m) 2m ago 12m 381M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:54:46.477 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (12m) 2m ago 12m 588M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:54:46.477 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (5m) 37s ago 15m 63.1M - 2.33.4 514e6a882f6e 193819b86dea 2024-02-10T06:54:46.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 261483 10.04 MB/sec execute 487 sec latency 65.027 ms 2024-02-10T06:54:47.032 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:54:47.033 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:54:47.033 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:54:47.033 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:54:47.033 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:54:47.033 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:54:47.033 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:54:47.033 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:54:47.033 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-10T06:54:47.033 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T06:54:47.033 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:54:47.033 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:54:47.033 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:54:47.033 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:54:47.033 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:54:47.034 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-10T06:54:47.034 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 7 2024-02-10T06:54:47.034 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:54:47.034 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:54:47.111 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:46 smithi008 ceph-mon[66415]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:54:47.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:46 smithi029 ceph-mon[53210]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:54:47.524 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:54:47.524 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:54:47.524 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:54:47.524 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:54:47.524 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:54:47.524 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:54:47.524 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-02-10T06:54:47.524 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-02-10T06:54:47.524 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:54:47.525 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "9/23 daemons upgraded", 2024-02-10T06:54:47.525 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:54:47.525 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:54:47.525 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:54:47.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 261763 10.03 MB/sec execute 488 sec latency 71.557 ms 2024-02-10T06:54:47.921 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:47 smithi008 podman[80401]: 2024-02-10 06:54:47.785581704 +0000 UTC m=+1.511370877 container remove 164f0c68d0cd73a496dea05b0d4f23cf4215a18b54e05569a219bc17b2585353 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-activate, RELEASE=main-290212a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-10T06:54:48.076 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 206/1683 objects degraded (12.240%), 27 pgs degraded 2024-02-10T06:54:48.076 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-10T06:54:48.076 INFO:teuthology.orchestra.run.smithi008.stdout: osd.3 (root=default,host=smithi008) is down 2024-02-10T06:54:48.077 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 206/1683 objects degraded (12.240%), 27 pgs degraded 2024-02-10T06:54:48.077 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1 is active+undersized+degraded, acting [2,0] 2024-02-10T06:54:48.077 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.9 is active+undersized+degraded, acting [1,7] 2024-02-10T06:54:48.077 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.a is active+undersized+degraded, acting [1,7] 2024-02-10T06:54:48.077 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.d is active+undersized+degraded, acting [1,4] 2024-02-10T06:54:48.077 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.e is active+undersized+degraded, acting [2,7] 2024-02-10T06:54:48.077 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.12 is active+undersized+degraded, acting [5,7] 2024-02-10T06:54:48.077 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.13 is active+undersized+degraded, acting [0,4] 2024-02-10T06:54:48.077 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.14 is active+undersized+degraded, acting [6,5] 2024-02-10T06:54:48.077 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.19 is active+undersized+degraded, acting [6,0] 2024-02-10T06:54:48.077 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1b is active+undersized+degraded, acting [7,6] 2024-02-10T06:54:48.077 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1e is active+undersized+degraded, acting [0,5] 2024-02-10T06:54:48.077 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1 is active+undersized+degraded, acting [0,4] 2024-02-10T06:54:48.077 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-02-10T06:54:48.078 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.5 is active+undersized+degraded, acting [5,2] 2024-02-10T06:54:48.078 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.7 is active+undersized+degraded, acting [7,0] 2024-02-10T06:54:48.078 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.8 is active+undersized+degraded, acting [1,7] 2024-02-10T06:54:48.078 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.b is active+undersized+degraded, acting [0,4] 2024-02-10T06:54:48.078 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.c is active+undersized+degraded, acting [5,6] 2024-02-10T06:54:48.078 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.12 is active+undersized+degraded, acting [0,7] 2024-02-10T06:54:48.078 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.15 is active+undersized+degraded, acting [7,4] 2024-02-10T06:54:48.078 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.17 is active+undersized+degraded, acting [0,5] 2024-02-10T06:54:48.078 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-02-10T06:54:48.078 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.19 is active+undersized+degraded, acting [1,4] 2024-02-10T06:54:48.078 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1e is active+undersized+degraded, acting [6,2] 2024-02-10T06:54:48.078 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-02-10T06:54:48.078 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.14 is active+undersized+degraded, acting [1,7] 2024-02-10T06:54:48.078 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.19 is active+undersized+degraded, acting [2,0] 2024-02-10T06:54:48.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:47 smithi008 ceph-mon[66415]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:54:48.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:47 smithi008 ceph-mon[66415]: pgmap v166: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 347 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 6.4 MiB/s wr, 621 op/s; 206/1683 objects degraded (12.240%) 2024-02-10T06:54:48.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:47 smithi008 ceph-mon[66415]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:54:48.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:47 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/1768515887' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:54:48.183 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:48 smithi008 podman[80522]: 2024-02-10 06:54:47.966113196 +0000 UTC m=+0.025811625 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:54:48.183 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:48 smithi008 podman[80522]: 2024-02-10 06:54:48.101451919 +0000 UTC m=+0.161150329 container create fc3683d167e9a87718b94f71357c557e4e1a2bd99891e0c957902f35c2a4ef36 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-10T06:54:48.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:47 smithi029 ceph-mon[53210]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:54:48.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:47 smithi029 ceph-mon[53210]: pgmap v166: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 347 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 6.4 MiB/s wr, 621 op/s; 206/1683 objects degraded (12.240%) 2024-02-10T06:54:48.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:47 smithi029 ceph-mon[53210]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:54:48.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:47 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/1768515887' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:54:48.444 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:48 smithi008 podman[80522]: 2024-02-10 06:54:48.363425851 +0000 UTC m=+0.423124276 container init fc3683d167e9a87718b94f71357c557e4e1a2bd99891e0c957902f35c2a4ef36 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, RELEASE=main-290212a, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131) 2024-02-10T06:54:48.444 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:48 smithi008 podman[80522]: 2024-02-10 06:54:48.371187896 +0000 UTC m=+0.430886317 container start fc3683d167e9a87718b94f71357c557e4e1a2bd99891e0c957902f35c2a4ef36 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:54:48.444 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:48 smithi008 ceph-osd[80543]: -- 172.21.15.8:0/3960285451 <== mon.0 v2:172.21.15.8:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a435bf71e0 con 0x55a435bc8c00 2024-02-10T06:54:48.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 262002 10.03 MB/sec execute 489 sec latency 56.225 ms 2024-02-10T06:54:48.698 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:48 smithi008 bash[80522]: fc3683d167e9a87718b94f71357c557e4e1a2bd99891e0c957902f35c2a4ef36 2024-02-10T06:54:48.698 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:48 smithi008 systemd[1]: Started Ceph osd.3 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:54:48.950 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:48 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3[80539]: 2024-02-10T06:54:48.941+0000 7f578ce9d7c0 -1 Falling back to public interface 2024-02-10T06:54:48.950 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:48 smithi008 ceph-mon[66415]: from='client.44253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:54:48.950 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:48 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/2956272444' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:54:48.950 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:48.950 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:54:48.951 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:48.951 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:48.951 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:54:49.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:48 smithi029 ceph-mon[53210]: from='client.44253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:54:49.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:48 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/2956272444' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:54:49.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:49.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:54:49.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:49.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:49.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:54:49.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 262233 10.01 MB/sec execute 490 sec latency 76.403 ms 2024-02-10T06:54:49.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:49 smithi008 ceph-mon[66415]: pgmap v167: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.3 MiB/s wr, 277 op/s; 206/1683 objects degraded (12.240%) 2024-02-10T06:54:50.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:49 smithi029 ceph-mon[53210]: pgmap v167: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.3 MiB/s wr, 277 op/s; 206/1683 objects degraded (12.240%) 2024-02-10T06:54:50.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 262573 10.02 MB/sec execute 491 sec latency 65.289 ms 2024-02-10T06:54:51.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:50 smithi008 ceph-mon[66415]: pgmap v168: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 340 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.6 MiB/s wr, 457 op/s; 197/1569 objects degraded (12.556%) 2024-02-10T06:54:51.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:50 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 197/1569 objects degraded (12.556%), 27 pgs degraded (PG_DEGRADED) 2024-02-10T06:54:51.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:50 smithi029 ceph-mon[53210]: pgmap v168: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 340 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.6 MiB/s wr, 457 op/s; 197/1569 objects degraded (12.556%) 2024-02-10T06:54:51.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:50 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 197/1569 objects degraded (12.556%), 27 pgs degraded (PG_DEGRADED) 2024-02-10T06:54:51.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 262884 10.01 MB/sec execute 492 sec latency 48.785 ms 2024-02-10T06:54:52.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 263241 10.02 MB/sec execute 493 sec latency 53.147 ms 2024-02-10T06:54:53.564 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:53 smithi008 ceph-mon[66415]: pgmap v169: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 340 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.2 MiB/s wr, 413 op/s; 197/1569 objects degraded (12.556%) 2024-02-10T06:54:53.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 263543 10.02 MB/sec execute 494 sec latency 44.783 ms 2024-02-10T06:54:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:53 smithi029 ceph-mon[53210]: pgmap v169: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 340 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.2 MiB/s wr, 413 op/s; 197/1569 objects degraded (12.556%) 2024-02-10T06:54:54.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 263833 10.02 MB/sec execute 495 sec latency 55.635 ms 2024-02-10T06:54:55.087 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:54 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3[80539]: 2024-02-10T06:54:54.947+0000 7f578ce9d7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-02-10T06:54:55.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 264074 10.01 MB/sec execute 496 sec latency 74.580 ms 2024-02-10T06:54:55.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:55 smithi008 ceph-mon[66415]: pgmap v170: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.5 MiB/s wr, 527 op/s; 202/1656 objects degraded (12.198%) 2024-02-10T06:54:55.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:55 smithi029 ceph-mon[53210]: pgmap v170: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 347 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.5 MiB/s wr, 527 op/s; 202/1656 objects degraded (12.198%) 2024-02-10T06:54:56.631 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:56 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3[80539]: 2024-02-10T06:54:56.555+0000 7f578ce9d7c0 -1 osd.3 100 log_to_monitors true 2024-02-10T06:54:56.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 264265 9.99 MB/sec execute 497 sec latency 41.141 ms 2024-02-10T06:54:56.886 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:56 smithi008 ceph-mon[66415]: from='osd.3 [v2:172.21.15.8:6826/1976418352,v1:172.21.15.8:6827/1976418352]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:54:56.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:56 smithi029 ceph-mon[53210]: from='osd.3 [v2:172.21.15.8:6826/1976418352,v1:172.21.15.8:6827/1976418352]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:54:57.153 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 06:54:57 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3[80539]: 2024-02-10T06:54:57.047+0000 7f577a2ed700 -1 osd.3 100 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:54:57.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 264516 9.98 MB/sec execute 498 sec latency 61.770 ms 2024-02-10T06:54:57.938 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:57 smithi008 ceph-mon[66415]: pgmap v171: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.9 MiB/s wr, 373 op/s; 205/1659 objects degraded (12.357%) 2024-02-10T06:54:57.938 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:57 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 205/1659 objects degraded (12.357%), 27 pgs degraded (PG_DEGRADED) 2024-02-10T06:54:57.938 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:57 smithi008 ceph-mon[66415]: from='osd.3 [v2:172.21.15.8:6826/1976418352,v1:172.21.15.8:6827/1976418352]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T06:54:57.938 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:57 smithi008 ceph-mon[66415]: failed to encode map e103 with expected crc 2024-02-10T06:54:57.938 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:57 smithi008 ceph-mon[66415]: osdmap e103: 8 total, 7 up, 8 in 2024-02-10T06:54:57.938 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:57 smithi008 ceph-mon[66415]: from='osd.3 [v2:172.21.15.8:6826/1976418352,v1:172.21.15.8:6827/1976418352]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T06:54:57.938 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:57 smithi008 ceph-mon[66415]: failed to encode map e103 with expected crc 2024-02-10T06:54:57.939 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:57 smithi008 ceph-mon[66415]: from='osd.3 [v2:172.21.15.8:6826/1976418352,v1:172.21.15.8:6827/1976418352]' entity='osd.3' 2024-02-10T06:54:57.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:57 smithi029 ceph-mon[53210]: pgmap v171: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.9 MiB/s wr, 373 op/s; 205/1659 objects degraded (12.357%) 2024-02-10T06:54:57.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:57 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 205/1659 objects degraded (12.357%), 27 pgs degraded (PG_DEGRADED) 2024-02-10T06:54:57.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:57 smithi029 ceph-mon[53210]: from='osd.3 [v2:172.21.15.8:6826/1976418352,v1:172.21.15.8:6827/1976418352]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T06:54:57.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:57 smithi029 ceph-mon[53210]: failed to encode map e103 with expected crc 2024-02-10T06:54:57.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:57 smithi029 ceph-mon[53210]: osdmap e103: 8 total, 7 up, 8 in 2024-02-10T06:54:57.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:57 smithi029 ceph-mon[53210]: from='osd.3 [v2:172.21.15.8:6826/1976418352,v1:172.21.15.8:6827/1976418352]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T06:54:57.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:57 smithi029 ceph-mon[53210]: failed to encode map e103 with expected crc 2024-02-10T06:54:57.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:57 smithi029 ceph-mon[53210]: from='osd.3 [v2:172.21.15.8:6826/1976418352,v1:172.21.15.8:6827/1976418352]' entity='osd.3' 2024-02-10T06:54:58.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 264649 9.97 MB/sec execute 499 sec latency 639.801 ms 2024-02-10T06:54:58.945 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:58 smithi008 ceph-mon[66415]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:54:58.945 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:58 smithi008 ceph-mon[66415]: osd.3 [v2:172.21.15.8:6826/1976418352,v1:172.21.15.8:6827/1976418352] boot 2024-02-10T06:54:58.946 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:58 smithi008 ceph-mon[66415]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T06:54:58.946 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:58 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:54:58.946 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:58 smithi008 ceph-mon[66415]: failed to encode map e104 with expected crc 2024-02-10T06:54:58.946 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:58 smithi008 ceph-mon[66415]: failed to encode map e104 with expected crc 2024-02-10T06:54:58.946 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:58 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:58.946 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:58 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:58.946 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:58 smithi008 ceph-mon[66415]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T06:54:58.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:58 smithi029 ceph-mon[53210]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:54:58.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:58 smithi029 ceph-mon[53210]: osd.3 [v2:172.21.15.8:6826/1976418352,v1:172.21.15.8:6827/1976418352] boot 2024-02-10T06:54:58.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:58 smithi029 ceph-mon[53210]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T06:54:58.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:58 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:54:58.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:58 smithi029 ceph-mon[53210]: failed to encode map e104 with expected crc 2024-02-10T06:54:58.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:58 smithi029 ceph-mon[53210]: failed to encode map e104 with expected crc 2024-02-10T06:54:58.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:58 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:58.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:58 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:58.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:58 smithi029 ceph-mon[53210]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T06:54:59.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 264726 9.96 MB/sec execute 500 sec latency 1639.852 ms 2024-02-10T06:54:59.950 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:59 smithi008 ceph-mon[66415]: failed to encode map e104 with expected crc 2024-02-10T06:54:59.950 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:59 smithi008 ceph-mon[66415]: pgmap v174: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 348 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.2 MiB/s wr, 271 op/s; 205/1659 objects degraded (12.357%) 2024-02-10T06:54:59.950 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:59 smithi008 ceph-mon[66415]: failed to encode map e105 with expected crc 2024-02-10T06:54:59.950 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:59 smithi008 ceph-mon[66415]: failed to encode map e105 with expected crc 2024-02-10T06:54:59.950 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:59 smithi008 ceph-mon[66415]: failed to encode map e105 with expected crc 2024-02-10T06:54:59.950 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:59 smithi008 ceph-mon[66415]: failed to encode map e105 with expected crc 2024-02-10T06:54:59.951 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:59 smithi008 ceph-mon[66415]: failed to encode map e105 with expected crc 2024-02-10T06:54:59.951 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:59 smithi008 ceph-mon[66415]: failed to encode map e105 with expected crc 2024-02-10T06:54:59.951 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:59 smithi008 ceph-mon[66415]: failed to encode map e105 with expected crc 2024-02-10T06:54:59.951 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:59 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:59.951 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:59 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:59.951 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:54:59 smithi008 ceph-mon[66415]: osdmap e106: 8 total, 8 up, 8 in 2024-02-10T06:54:59.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:59 smithi029 ceph-mon[53210]: failed to encode map e104 with expected crc 2024-02-10T06:54:59.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:59 smithi029 ceph-mon[53210]: pgmap v174: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 348 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.2 MiB/s wr, 271 op/s; 205/1659 objects degraded (12.357%) 2024-02-10T06:54:59.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:59 smithi029 ceph-mon[53210]: failed to encode map e105 with expected crc 2024-02-10T06:54:59.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:59 smithi029 ceph-mon[53210]: failed to encode map e105 with expected crc 2024-02-10T06:54:59.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:59 smithi029 ceph-mon[53210]: failed to encode map e105 with expected crc 2024-02-10T06:54:59.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:59 smithi029 ceph-mon[53210]: failed to encode map e105 with expected crc 2024-02-10T06:54:59.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:59 smithi029 ceph-mon[53210]: failed to encode map e105 with expected crc 2024-02-10T06:54:59.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:59 smithi029 ceph-mon[53210]: failed to encode map e105 with expected crc 2024-02-10T06:54:59.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:59 smithi029 ceph-mon[53210]: failed to encode map e105 with expected crc 2024-02-10T06:54:59.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:59 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:59.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:59 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:54:59.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:54:59 smithi029 ceph-mon[53210]: osdmap e106: 8 total, 8 up, 8 in 2024-02-10T06:55:00.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 265046 9.96 MB/sec execute 501 sec latency 1771.945 ms 2024-02-10T06:55:00.959 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:00 smithi008 ceph-mon[66415]: failed to encode map e106 with expected crc 2024-02-10T06:55:00.959 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:00 smithi008 ceph-mon[66415]: failed to encode map e106 with expected crc 2024-02-10T06:55:00.960 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:00 smithi008 ceph-mon[66415]: failed to encode map e106 with expected crc 2024-02-10T06:55:00.960 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:00 smithi008 ceph-mon[66415]: failed to encode map e106 with expected crc 2024-02-10T06:55:00.960 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:00 smithi008 ceph-mon[66415]: failed to encode map e106 with expected crc 2024-02-10T06:55:01.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:00 smithi029 ceph-mon[53210]: failed to encode map e106 with expected crc 2024-02-10T06:55:01.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:00 smithi029 ceph-mon[53210]: failed to encode map e106 with expected crc 2024-02-10T06:55:01.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:00 smithi029 ceph-mon[53210]: failed to encode map e106 with expected crc 2024-02-10T06:55:01.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:00 smithi029 ceph-mon[53210]: failed to encode map e106 with expected crc 2024-02-10T06:55:01.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:00 smithi029 ceph-mon[53210]: failed to encode map e106 with expected crc 2024-02-10T06:55:01.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 265263 9.94 MB/sec execute 502 sec latency 897.574 ms 2024-02-10T06:55:01.952 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:01 smithi008 ceph-mon[66415]: pgmap v177: 97 pgs: 74 active+clean, 5 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 1 active+recovering+degraded, 3 remapped+peering; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.5 MiB/s wr, 390 op/s; 116/1542 objects degraded (7.523%); 71 KiB/s, 9 objects/s recovering 2024-02-10T06:55:01.952 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:01 smithi008 ceph-mon[66415]: failed to encode map e107 with expected crc 2024-02-10T06:55:01.952 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:01 smithi008 ceph-mon[66415]: osdmap e107: 8 total, 8 up, 8 in 2024-02-10T06:55:01.952 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:01 smithi008 ceph-mon[66415]: failed to encode map e107 with expected crc 2024-02-10T06:55:01.952 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:01 smithi008 ceph-mon[66415]: failed to encode map e107 with expected crc 2024-02-10T06:55:01.952 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:01 smithi008 ceph-mon[66415]: failed to encode map e107 with expected crc 2024-02-10T06:55:01.952 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:01 smithi008 ceph-mon[66415]: failed to encode map e107 with expected crc 2024-02-10T06:55:01.953 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:01 smithi008 ceph-mon[66415]: failed to encode map e107 with expected crc 2024-02-10T06:55:01.953 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:01 smithi008 ceph-mon[66415]: failed to encode map e107 with expected crc 2024-02-10T06:55:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:01 smithi029 ceph-mon[53210]: pgmap v177: 97 pgs: 74 active+clean, 5 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 1 active+recovering+degraded, 3 remapped+peering; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.5 MiB/s wr, 390 op/s; 116/1542 objects degraded (7.523%); 71 KiB/s, 9 objects/s recovering 2024-02-10T06:55:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:01 smithi029 ceph-mon[53210]: failed to encode map e107 with expected crc 2024-02-10T06:55:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:01 smithi029 ceph-mon[53210]: osdmap e107: 8 total, 8 up, 8 in 2024-02-10T06:55:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:01 smithi029 ceph-mon[53210]: failed to encode map e107 with expected crc 2024-02-10T06:55:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:01 smithi029 ceph-mon[53210]: failed to encode map e107 with expected crc 2024-02-10T06:55:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:01 smithi029 ceph-mon[53210]: failed to encode map e107 with expected crc 2024-02-10T06:55:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:01 smithi029 ceph-mon[53210]: failed to encode map e107 with expected crc 2024-02-10T06:55:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:01 smithi029 ceph-mon[53210]: failed to encode map e107 with expected crc 2024-02-10T06:55:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:01 smithi029 ceph-mon[53210]: failed to encode map e107 with expected crc 2024-02-10T06:55:02.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 265651 9.94 MB/sec execute 503 sec latency 969.124 ms 2024-02-10T06:55:02.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:02 smithi008 ceph-mon[66415]: failed to encode map e108 with expected crc 2024-02-10T06:55:02.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:02 smithi008 ceph-mon[66415]: osdmap e108: 8 total, 8 up, 8 in 2024-02-10T06:55:02.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:02 smithi008 ceph-mon[66415]: failed to encode map e108 with expected crc 2024-02-10T06:55:02.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:02 smithi008 ceph-mon[66415]: failed to encode map e108 with expected crc 2024-02-10T06:55:02.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:02 smithi008 ceph-mon[66415]: failed to encode map e108 with expected crc 2024-02-10T06:55:02.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:02 smithi008 ceph-mon[66415]: failed to encode map e108 with expected crc 2024-02-10T06:55:03.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:02 smithi029 ceph-mon[53210]: failed to encode map e108 with expected crc 2024-02-10T06:55:03.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:02 smithi029 ceph-mon[53210]: osdmap e108: 8 total, 8 up, 8 in 2024-02-10T06:55:03.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:02 smithi029 ceph-mon[53210]: failed to encode map e108 with expected crc 2024-02-10T06:55:03.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:02 smithi029 ceph-mon[53210]: failed to encode map e108 with expected crc 2024-02-10T06:55:03.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:02 smithi029 ceph-mon[53210]: failed to encode map e108 with expected crc 2024-02-10T06:55:03.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:02 smithi029 ceph-mon[53210]: failed to encode map e108 with expected crc 2024-02-10T06:55:03.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 265960 9.94 MB/sec execute 504 sec latency 40.492 ms 2024-02-10T06:55:04.001 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:03 smithi008 ceph-mon[66415]: pgmap v180: 97 pgs: 89 active+clean, 1 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active, 1 active+recovering+degraded, 1 remapped+peering; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.4 MiB/s wr, 551 op/s; 59/1521 objects degraded (3.879%); 288 KiB/s, 35 objects/s recovering 2024-02-10T06:55:04.001 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:03 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 59/1521 objects degraded (3.879%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T06:55:04.001 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:04.001 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:55:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:03 smithi029 ceph-mon[53210]: pgmap v180: 97 pgs: 89 active+clean, 1 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active, 1 active+recovering+degraded, 1 remapped+peering; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.4 MiB/s wr, 551 op/s; 59/1521 objects degraded (3.879%); 288 KiB/s, 35 objects/s recovering 2024-02-10T06:55:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:03 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 59/1521 objects degraded (3.879%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T06:55:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:55:04.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 266313 9.94 MB/sec execute 505 sec latency 40.235 ms 2024-02-10T06:55:05.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 266668 9.95 MB/sec execute 506 sec latency 38.189 ms 2024-02-10T06:55:05.774 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:05 smithi008 ceph-mon[66415]: pgmap v181: 97 pgs: 94 active+clean, 2 active+recovery_wait+undersized+degraded+remapped, 1 active; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 4.6 MiB/s wr, 485 op/s; 42/1590 objects degraded (2.642%); 212 KiB/s, 31 objects/s recovering 2024-02-10T06:55:05.774 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:05.774 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:05.774 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:55:05.775 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:55:05.775 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:05.775 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:05.775 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:55:05.775 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:55:05.775 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:55:05.775 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:55:05.775 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T06:55:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:05 smithi029 ceph-mon[53210]: pgmap v181: 97 pgs: 94 active+clean, 2 active+recovery_wait+undersized+degraded+remapped, 1 active; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 4.6 MiB/s wr, 485 op/s; 42/1590 objects degraded (2.642%); 212 KiB/s, 31 objects/s recovering 2024-02-10T06:55:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:55:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:55:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:55:06.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:55:06.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:55:06.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:55:06.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T06:55:06.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 266967 9.95 MB/sec execute 507 sec latency 41.980 ms 2024-02-10T06:55:06.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:06 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T06:55:06.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:06 smithi008 ceph-mon[66415]: Upgrade: osd.4 is safe to restart 2024-02-10T06:55:06.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:06 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T06:55:06.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:06 smithi029 ceph-mon[53210]: Upgrade: osd.4 is safe to restart 2024-02-10T06:55:07.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 267239 9.95 MB/sec execute 508 sec latency 53.998 ms 2024-02-10T06:55:07.818 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:07 smithi008 ceph-mon[66415]: pgmap v182: 97 pgs: 96 active+clean, 1 active; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.4 MiB/s wr, 216 op/s; 268 KiB/s, 29 objects/s recovering 2024-02-10T06:55:07.818 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:07 smithi008 ceph-mon[66415]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/1590 objects degraded (2.642%), 2 pgs degraded) 2024-02-10T06:55:07.818 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:07 smithi008 ceph-mon[66415]: Cluster is now healthy 2024-02-10T06:55:07.818 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:07 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:07.818 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:07 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T06:55:07.818 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:07 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:55:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:07 smithi029 ceph-mon[53210]: pgmap v182: 97 pgs: 96 active+clean, 1 active; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.4 MiB/s wr, 216 op/s; 268 KiB/s, 29 objects/s recovering 2024-02-10T06:55:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:07 smithi029 ceph-mon[53210]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/1590 objects degraded (2.642%), 2 pgs degraded) 2024-02-10T06:55:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:07 smithi029 ceph-mon[53210]: Cluster is now healthy 2024-02-10T06:55:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:07 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:07 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T06:55:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:07 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:55:08.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 267512 9.94 MB/sec execute 509 sec latency 102.627 ms 2024-02-10T06:55:08.900 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:08 smithi008 ceph-mon[66415]: Upgrade: Updating osd.4 2024-02-10T06:55:08.900 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:08 smithi008 ceph-mon[66415]: Deploying daemon osd.4 on smithi029 2024-02-10T06:55:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:08 smithi029 ceph-mon[53210]: Upgrade: Updating osd.4 2024-02-10T06:55:09.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:08 smithi029 ceph-mon[53210]: Deploying daemon osd.4 on smithi029 2024-02-10T06:55:09.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 267744 9.92 MB/sec execute 510 sec latency 133.195 ms 2024-02-10T06:55:09.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:09 smithi008 ceph-mon[66415]: pgmap v183: 97 pgs: 97 active+clean; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.7 MiB/s wr, 267 op/s; 196 KiB/s, 22 objects/s recovering 2024-02-10T06:55:10.040 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:09 smithi029 ceph-mon[53210]: pgmap v183: 97 pgs: 97 active+clean; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.7 MiB/s wr, 267 op/s; 196 KiB/s, 22 objects/s recovering 2024-02-10T06:55:10.670 INFO:teuthology.orchestra.run.smithi008.stdout: 5 267943 9.91 MB/sec execute 511 sec latency 55.707 ms 2024-02-10T06:55:11.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 268221 9.91 MB/sec execute 512 sec latency 48.516 ms 2024-02-10T06:55:11.820 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:11 smithi008 ceph-mon[66415]: pgmap v184: 97 pgs: 97 active+clean; 350 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.4 MiB/s wr, 447 op/s; 163 KiB/s, 18 objects/s recovering 2024-02-10T06:55:11.885 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:11 smithi029 ceph-mon[53210]: pgmap v184: 97 pgs: 97 active+clean; 350 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.4 MiB/s wr, 447 op/s; 163 KiB/s, 18 objects/s recovering 2024-02-10T06:55:12.208 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:12 smithi029 systemd[1]: Stopping Ceph osd.4 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:55:12.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 268539 9.90 MB/sec execute 513 sec latency 78.720 ms 2024-02-10T06:55:12.708 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:12 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[31321]: 2024-02-10T06:55:12.388+0000 7fd63e4c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:55:12.708 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:12 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[31321]: 2024-02-10T06:55:12.388+0000 7fd63e4c2700 -1 osd.4 108 *** Got signal Terminated *** 2024-02-10T06:55:12.708 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:12 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[31321]: 2024-02-10T06:55:12.388+0000 7fd63e4c2700 -1 osd.4 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:55:13.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:12 smithi008 ceph-mon[66415]: osd.4 marked itself down and dead 2024-02-10T06:55:13.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:12 smithi029 ceph-mon[53210]: osd.4 marked itself down and dead 2024-02-10T06:55:13.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 268625 9.89 MB/sec execute 514 sec latency 867.416 ms 2024-02-10T06:55:13.699 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:13 smithi029 podman[57696]: 2024-02-10 06:55:13.410651557 +0000 UTC m=+1.163493743 container died a32c1bdd2400cdf239f99d0717242ae490b66ee714177e2971c6c6c7a6044c88 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4, RELEASE=HEAD, com.redhat.component=centos-stream-container, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, version=8, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux ) 2024-02-10T06:55:13.958 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:13 smithi029 podman[57696]: 2024-02-10 06:55:13.791105006 +0000 UTC m=+1.543947194 container cleanup a32c1bdd2400cdf239f99d0717242ae490b66ee714177e2971c6c6c7a6044c88 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, release=754, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, GIT_CLEAN=True, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, vcs-type=git, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, ceph=True, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , distribution-scope=public, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870) 2024-02-10T06:55:13.958 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:13 smithi029 bash[57696]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4 2024-02-10T06:55:13.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:13 smithi029 ceph-mon[53210]: pgmap v185: 97 pgs: 97 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.2 MiB/s wr, 412 op/s; 148 KiB/s, 17 objects/s recovering 2024-02-10T06:55:13.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:13 smithi029 ceph-mon[53210]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:55:13.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:13 smithi029 ceph-mon[53210]: failed to encode map e109 with expected crc 2024-02-10T06:55:13.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:13 smithi029 ceph-mon[53210]: osdmap e109: 8 total, 7 up, 8 in 2024-02-10T06:55:13.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:13 smithi029 ceph-mon[53210]: failed to encode map e109 with expected crc 2024-02-10T06:55:13.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:13 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:14.022 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:13 smithi008 ceph-mon[66415]: pgmap v185: 97 pgs: 97 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.2 MiB/s wr, 412 op/s; 148 KiB/s, 17 objects/s recovering 2024-02-10T06:55:14.022 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:13 smithi008 ceph-mon[66415]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:55:14.022 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:13 smithi008 ceph-mon[66415]: failed to encode map e109 with expected crc 2024-02-10T06:55:14.022 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:13 smithi008 ceph-mon[66415]: osdmap e109: 8 total, 7 up, 8 in 2024-02-10T06:55:14.022 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:13 smithi008 ceph-mon[66415]: failed to encode map e109 with expected crc 2024-02-10T06:55:14.022 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:13 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:14.410 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:14 smithi029 podman[57715]: 2024-02-10 06:55:14.095620184 +0000 UTC m=+0.681645189 container remove a32c1bdd2400cdf239f99d0717242ae490b66ee714177e2971c6c6c7a6044c88 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4, release=754, name=centos-stream, GIT_CLEAN=True, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, RELEASE=HEAD, io.buildah.version=1.19.8, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vcs-type=git, io.openshift.expose-services=, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-10T06:55:14.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 268936 9.88 MB/sec execute 515 sec latency 1023.115 ms 2024-02-10T06:55:14.708 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:14 smithi029 podman[57805]: 2024-02-10 06:55:14.312703356 +0000 UTC m=+0.017938973 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:55:14.708 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:14 smithi029 podman[57805]: 2024-02-10 06:55:14.422650402 +0000 UTC m=+0.127886009 container create 172c8075e005803afbf0f0fd52cfdceb7ab123333ddfef13721e7ebb8991d576 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-10T06:55:15.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:14 smithi008 ceph-mon[66415]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:14 smithi008 ceph-mon[66415]: osdmap e110: 8 total, 7 up, 8 in 2024-02-10T06:55:15.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:14 smithi008 ceph-mon[66415]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:14 smithi008 ceph-mon[66415]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:14 smithi008 ceph-mon[66415]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:14 smithi008 ceph-mon[66415]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:14 smithi008 ceph-mon[66415]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:14 smithi008 ceph-mon[66415]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:14 smithi008 ceph-mon[66415]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:14 smithi008 ceph-mon[66415]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.056 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:14 smithi029 ceph-mon[53210]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.056 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:14 smithi029 ceph-mon[53210]: osdmap e110: 8 total, 7 up, 8 in 2024-02-10T06:55:15.056 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:14 smithi029 ceph-mon[53210]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.056 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:14 smithi029 ceph-mon[53210]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.057 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:14 smithi029 ceph-mon[53210]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.057 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:14 smithi029 ceph-mon[53210]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.057 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:14 smithi029 ceph-mon[53210]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.057 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:14 smithi029 ceph-mon[53210]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.057 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:14 smithi029 ceph-mon[53210]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.057 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:14 smithi029 ceph-mon[53210]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.057 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:14 smithi029 podman[57805]: 2024-02-10 06:55:14.736043566 +0000 UTC m=+0.441279193 container init 172c8075e005803afbf0f0fd52cfdceb7ab123333ddfef13721e7ebb8991d576 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.build-date=20240131) 2024-02-10T06:55:15.057 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:14 smithi029 podman[57805]: 2024-02-10 06:55:14.740286577 +0000 UTC m=+0.445522184 container start 172c8075e005803afbf0f0fd52cfdceb7ab123333ddfef13721e7ebb8991d576 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-290212a, io.buildah.version=1.33.2) 2024-02-10T06:55:15.057 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:14 smithi029 podman[57805]: 2024-02-10 06:55:14.846872103 +0000 UTC m=+0.552107729 container attach 172c8075e005803afbf0f0fd52cfdceb7ab123333ddfef13721e7ebb8991d576 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-10T06:55:15.436 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:15 smithi029 podman[57805]: 2024-02-10 06:55:15.056761301 +0000 UTC m=+0.761996899 container died 172c8075e005803afbf0f0fd52cfdceb7ab123333ddfef13721e7ebb8991d576 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, RELEASE=main-290212a, CEPH_POINT_RELEASE=) 2024-02-10T06:55:15.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 269246 9.88 MB/sec execute 516 sec latency 54.058 ms 2024-02-10T06:55:15.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:15 smithi008 ceph-mon[66415]: failed to encode map e110 with expected crc 2024-02-10T06:55:15.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:15 smithi008 ceph-mon[66415]: pgmap v188: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 7 stale+active+clean, 65 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.2 MiB/s wr, 565 op/s; 151/1518 objects degraded (9.947%) 2024-02-10T06:55:15.789 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:15 smithi008 ceph-mon[66415]: Health check failed: Degraded data redundancy: 151/1518 objects degraded (9.947%), 19 pgs degraded (PG_DEGRADED) 2024-02-10T06:55:16.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:15 smithi029 ceph-mon[53210]: failed to encode map e110 with expected crc 2024-02-10T06:55:16.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:15 smithi029 ceph-mon[53210]: pgmap v188: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 7 stale+active+clean, 65 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.2 MiB/s wr, 565 op/s; 151/1518 objects degraded (9.947%) 2024-02-10T06:55:16.035 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:15 smithi029 ceph-mon[53210]: Health check failed: Degraded data redundancy: 151/1518 objects degraded (9.947%), 19 pgs degraded (PG_DEGRADED) 2024-02-10T06:55:16.035 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:15 smithi029 podman[57831]: 2024-02-10 06:55:15.782686117 +0000 UTC m=+0.720532248 container remove 172c8075e005803afbf0f0fd52cfdceb7ab123333ddfef13721e7ebb8991d576 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-02-10T06:55:16.035 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:15 smithi029 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.4.service: Deactivated successfully. 2024-02-10T06:55:16.036 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:15 smithi029 systemd[1]: Stopped Ceph osd.4 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:55:16.036 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:15 smithi029 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.4.service: Consumed 5min 13.110s CPU time. 2024-02-10T06:55:16.327 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:16 smithi029 systemd[1]: Starting Ceph osd.4 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:55:16.327 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:16 smithi029 podman[57932]: 2024-02-10 06:55:16.179437436 +0000 UTC m=+0.021761237 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:55:16.328 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:16 smithi029 podman[57932]: 2024-02-10 06:55:16.327594067 +0000 UTC m=+0.169917846 container create 459822c1e851a9abfc27f0f4b64dc4b84cdee71c3af5cb55fcb4127ef8e3e7ab (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-activate, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-10T06:55:16.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 269534 9.87 MB/sec execute 517 sec latency 54.851 ms 2024-02-10T06:55:16.958 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:16 smithi029 podman[57932]: 2024-02-10 06:55:16.584483511 +0000 UTC m=+0.426807296 container init 459822c1e851a9abfc27f0f4b64dc4b84cdee71c3af5cb55fcb4127ef8e3e7ab (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-290212a) 2024-02-10T06:55:16.958 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:16 smithi029 podman[57932]: 2024-02-10 06:55:16.588787365 +0000 UTC m=+0.431111142 container start 459822c1e851a9abfc27f0f4b64dc4b84cdee71c3af5cb55fcb4127ef8e3e7ab (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-activate, ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:55:16.958 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:16 smithi029 podman[57932]: 2024-02-10 06:55:16.702080402 +0000 UTC m=+0.544404203 container attach 459822c1e851a9abfc27f0f4b64dc4b84cdee71c3af5cb55fcb4127ef8e3e7ab (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-activate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:55:17.458 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:17 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-activate[57947]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T06:55:17.458 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:17 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-activate[57947]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-10T06:55:17.458 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:17 smithi029 bash[57932]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T06:55:17.458 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:17 smithi029 bash[57932]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-10T06:55:17.458 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:17 smithi029 bash[57932]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-10T06:55:17.458 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:17 smithi029 bash[57932]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T06:55:17.459 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:17 smithi029 bash[57932]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-10T06:55:17.459 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:17 smithi029 bash[57932]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T06:55:17.459 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:17 smithi029 bash[57932]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-10T06:55:17.459 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:17 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-activate[57947]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-10T06:55:17.459 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:17 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-activate[57947]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T06:55:17.459 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:17 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-activate[57947]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-10T06:55:17.459 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:17 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-activate[57947]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T06:55:17.459 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:17 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-activate[57947]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-10T06:55:17.459 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:17 smithi029 podman[57932]: 2024-02-10 06:55:17.175459188 +0000 UTC m=+1.017782961 container died 459822c1e851a9abfc27f0f4b64dc4b84cdee71c3af5cb55fcb4127ef8e3e7ab (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-290212a, ceph=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-02-10T06:55:17.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 269851 9.87 MB/sec execute 518 sec latency 53.680 ms 2024-02-10T06:55:17.824 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:17 smithi008 ceph-mon[66415]: pgmap v189: 97 pgs: 7 active+undersized, 26 active+undersized+degraded, 5 stale+active+clean, 59 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.6 MiB/s wr, 480 op/s; 204/1497 objects degraded (13.627%) 2024-02-10T06:55:18.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:17 smithi029 ceph-mon[53210]: pgmap v189: 97 pgs: 7 active+undersized, 26 active+undersized+degraded, 5 stale+active+clean, 59 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.6 MiB/s wr, 480 op/s; 204/1497 objects degraded (13.627%) 2024-02-10T06:55:18.474 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:18 smithi029 podman[58063]: 2024-02-10 06:55:18.213689293 +0000 UTC m=+1.032708569 container remove 459822c1e851a9abfc27f0f4b64dc4b84cdee71c3af5cb55fcb4127ef8e3e7ab (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:55:18.474 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:18 smithi029 podman[58124]: 2024-02-10 06:55:18.352106066 +0000 UTC m=+0.021420932 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:55:18.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 270162 9.87 MB/sec execute 519 sec latency 50.590 ms 2024-02-10T06:55:18.684 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:55:18.731 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:18 smithi029 podman[58124]: 2024-02-10 06:55:18.474483249 +0000 UTC m=+0.143798094 container create 2b1f7cbc27ce22de862cb2cb319f02b0da2e1695516a5e6f6bdb7f4f12e489d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-290212a, ceph=True) 2024-02-10T06:55:19.208 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:18 smithi029 podman[58124]: 2024-02-10 06:55:18.731283466 +0000 UTC m=+0.400598317 container init 2b1f7cbc27ce22de862cb2cb319f02b0da2e1695516a5e6f6bdb7f4f12e489d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-10T06:55:19.208 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:18 smithi029 podman[58124]: 2024-02-10 06:55:18.734579435 +0000 UTC m=+0.403894282 container start 2b1f7cbc27ce22de862cb2cb319f02b0da2e1695516a5e6f6bdb7f4f12e489d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-290212a, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:55:19.208 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:18 smithi029 ceph-osd[58143]: -- 172.21.15.29:0/2344177004 <== mon.0 v2:172.21.15.8:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5596781d31e0 con 0x5596781a5000 2024-02-10T06:55:19.209 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:18 smithi029 bash[58124]: 2b1f7cbc27ce22de862cb2cb319f02b0da2e1695516a5e6f6bdb7f4f12e489d1 2024-02-10T06:55:19.209 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:18 smithi029 systemd[1]: Started Ceph osd.4 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:55:19.573 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:55:19.573 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (7m) 21s ago 17m 15.6M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:55:19.573 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (3m) 21s ago 17m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T06:55:19.573 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (3m) 3m ago 16m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T06:55:19.573 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (7m) 21s ago 17m 43.0M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:55:19.574 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (11m) 21s ago 11m 38.5M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:55:19.574 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (11m) 3m ago 11m 14.5M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:55:19.574 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (6m) 21s ago 18m 507M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:55:19.574 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (5m) 3m ago 16m 449M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:55:19.574 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (5m) 21s ago 18m 53.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:55:19.574 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (4m) 3m ago 16m 35.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:55:19.574 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (10m) 21s ago 10m 271M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:55:19.574 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (10m) 3m ago 10m 84.3M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:55:19.574 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (7m) 21s ago 17m 11.4M - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:55:19.574 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (6m) 3m ago 16m 18.7M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:55:19.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (2m) 21s ago 15m 355M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 acb0107eae6c 2024-02-10T06:55:19.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (2m) 21s ago 15m 282M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 09363f1c35ae 2024-02-10T06:55:19.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (80s) 21s ago 14m 237M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 bd6bbb022085 2024-02-10T06:55:19.574 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (31s) 21s ago 14m 15.0M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc3683d167e9 2024-02-10T06:55:19.575 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 starting - - - 2705M 2024-02-10T06:55:19.575 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (13m) 3m ago 13m 461M 2705M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:55:19.575 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (13m) 3m ago 13m 381M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:55:19.575 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (12m) 3m ago 12m 588M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:55:19.575 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (5m) 21s ago 16m 63.3M - 2.33.4 514e6a882f6e 193819b86dea 2024-02-10T06:55:19.608 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:19 smithi008 ceph-mon[66415]: pgmap v190: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 MiB/s wr, 228 op/s; 234/1524 objects degraded (15.354%) 2024-02-10T06:55:19.608 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:19 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:19.608 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:19 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:55:19.608 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:19 smithi008 ceph-mon[66415]: from='client.34360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:55:19.608 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:19 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:19.608 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:19 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:19.608 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:19 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:55:19.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 270479 9.88 MB/sec execute 520 sec latency 45.636 ms 2024-02-10T06:55:19.708 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:19 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T06:55:19.275+0000 7fa5687f07c0 -1 Falling back to public interface 2024-02-10T06:55:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:19 smithi029 ceph-mon[53210]: pgmap v190: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 MiB/s wr, 228 op/s; 234/1524 objects degraded (15.354%) 2024-02-10T06:55:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:19 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:19 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:55:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:19 smithi029 ceph-mon[53210]: from='client.34360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:55:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:19 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:19 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:19 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:55:20.184 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:55:20.185 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:55:20.185 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:55:20.185 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:55:20.185 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:55:20.185 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:55:20.185 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:55:20.185 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:55:20.185 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-02-10T06:55:20.185 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 4 2024-02-10T06:55:20.185 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:55:20.185 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:55:20.185 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:55:20.186 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:55:20.186 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:55:20.186 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-02-10T06:55:20.186 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T06:55:20.186 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:55:20.186 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:55:20.369 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:20 smithi008 ceph-mon[66415]: from='client.34366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:55:20.369 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:20 smithi008 ceph-mon[66415]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:55:20.369 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:20 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/573176369' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:55:20.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 270770 9.88 MB/sec execute 521 sec latency 44.878 ms 2024-02-10T06:55:20.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:20 smithi029 ceph-mon[53210]: from='client.34366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:55:20.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:20 smithi029 ceph-mon[53210]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:55:20.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:20 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/573176369' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:55:20.714 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:55:20.714 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:55:20.714 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:55:20.714 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:55:20.714 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:55:20.714 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:55:20.714 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-02-10T06:55:20.714 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-02-10T06:55:20.714 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:55:20.715 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "10/23 daemons upgraded", 2024-02-10T06:55:20.715 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:55:20.715 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:55:20.715 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:55:21.384 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:21 smithi008 ceph-mon[66415]: pgmap v191: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 4.8 MiB/s wr, 503 op/s; 255/1602 objects degraded (15.918%) 2024-02-10T06:55:21.384 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:21 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 255/1602 objects degraded (15.918%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T06:55:21.384 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:21 smithi008 ceph-mon[66415]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:55:21.426 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 255/1602 objects degraded (15.918%), 28 pgs degraded 2024-02-10T06:55:21.427 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-10T06:55:21.427 INFO:teuthology.orchestra.run.smithi008.stdout: osd.4 (root=default,host=smithi029) is down 2024-02-10T06:55:21.427 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 255/1602 objects degraded (15.918%), 28 pgs degraded 2024-02-10T06:55:21.427 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.5 is active+undersized+degraded, acting [7,0] 2024-02-10T06:55:21.427 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.6 is active+undersized+degraded, acting [1,6] 2024-02-10T06:55:21.427 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.b is active+undersized+degraded, acting [7,5] 2024-02-10T06:55:21.427 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.d is active+undersized+degraded, acting [1,3] 2024-02-10T06:55:21.427 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.f is active+undersized+degraded, acting [0,7] 2024-02-10T06:55:21.427 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.11 is active+undersized+degraded, acting [6,1] 2024-02-10T06:55:21.427 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.13 is active+undersized+degraded, acting [0,3] 2024-02-10T06:55:21.427 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.18 is active+undersized+degraded, acting [5,7] 2024-02-10T06:55:21.428 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1a is active+undersized+degraded, acting [6,7] 2024-02-10T06:55:21.428 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1f is active+undersized+degraded, acting [0,7] 2024-02-10T06:55:21.428 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1 is active+undersized+degraded, acting [0,3] 2024-02-10T06:55:21.428 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-02-10T06:55:21.428 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.6 is active+undersized+degraded, acting [0,1] 2024-02-10T06:55:21.428 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-02-10T06:55:21.428 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.a is active+undersized+degraded, acting [6,1] 2024-02-10T06:55:21.428 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.b is active+undersized+degraded, acting [3,0] 2024-02-10T06:55:21.429 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.e is active+undersized+degraded, acting [7,1] 2024-02-10T06:55:21.429 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.f is active+undersized+degraded, acting [7,0] 2024-02-10T06:55:21.429 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.11 is active+undersized+degraded, acting [7,6] 2024-02-10T06:55:21.429 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.13 is active+undersized+degraded, acting [7,2] 2024-02-10T06:55:21.429 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-02-10T06:55:21.429 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.15 is active+undersized+degraded, acting [7,3] 2024-02-10T06:55:21.429 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-02-10T06:55:21.429 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-02-10T06:55:21.429 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1b is active+undersized+degraded, acting [0,7] 2024-02-10T06:55:21.429 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-02-10T06:55:21.429 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-02-10T06:55:21.429 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.5 is active+undersized+degraded, acting [6,7] 2024-02-10T06:55:21.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 271108 9.87 MB/sec execute 522 sec latency 57.200 ms 2024-02-10T06:55:21.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:21 smithi029 ceph-mon[53210]: pgmap v191: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 4.8 MiB/s wr, 503 op/s; 255/1602 objects degraded (15.918%) 2024-02-10T06:55:21.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:21 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 255/1602 objects degraded (15.918%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T06:55:21.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:21 smithi029 ceph-mon[53210]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:55:22.393 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:22 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/2757515019' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:55:22.643 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:22 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/2757515019' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:55:22.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 271343 9.86 MB/sec execute 523 sec latency 42.228 ms 2024-02-10T06:55:23.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:23 smithi008 ceph-mon[66415]: pgmap v192: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 346 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.3 MiB/s wr, 433 op/s; 258/1614 objects degraded (15.985%) 2024-02-10T06:55:23.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:23 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:23.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:23 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:23.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 271591 9.85 MB/sec execute 524 sec latency 52.117 ms 2024-02-10T06:55:23.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:23 smithi029 ceph-mon[53210]: pgmap v192: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 346 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.3 MiB/s wr, 433 op/s; 258/1614 objects degraded (15.985%) 2024-02-10T06:55:23.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:23 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:23.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:23 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:24.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 271863 9.85 MB/sec execute 525 sec latency 46.551 ms 2024-02-10T06:55:25.426 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:25 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T06:55:25.293+0000 7fa5687f07c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-02-10T06:55:25.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 272169 9.84 MB/sec execute 526 sec latency 62.330 ms 2024-02-10T06:55:25.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:25 smithi029 ceph-mon[53210]: pgmap v193: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 351 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 5.2 MiB/s wr, 532 op/s; 261/1629 objects degraded (16.022%) 2024-02-10T06:55:25.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:25 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:25.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:25 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:25.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:25 smithi008 ceph-mon[66415]: pgmap v193: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 351 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 5.2 MiB/s wr, 532 op/s; 261/1629 objects degraded (16.022%) 2024-02-10T06:55:25.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:25 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:25.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:25 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:26.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 272559 9.84 MB/sec execute 527 sec latency 51.969 ms 2024-02-10T06:55:26.738 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:26 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 261/1623 objects degraded (16.081%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T06:55:26.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:26 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 261/1623 objects degraded (16.081%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T06:55:27.458 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:27 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T06:55:27.112+0000 7fa5687f07c0 -1 osd.4 108 log_to_monitors true 2024-02-10T06:55:27.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 272811 9.83 MB/sec execute 528 sec latency 149.575 ms 2024-02-10T06:55:27.821 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:27 smithi008 ceph-mon[66415]: pgmap v194: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 350 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.9 MiB/s wr, 357 op/s; 261/1623 objects degraded (16.081%) 2024-02-10T06:55:27.821 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:27 smithi008 ceph-mon[66415]: from='osd.4 [v2:172.21.15.29:6800/3889378905,v1:172.21.15.29:6801/3889378905]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:55:27.821 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:27 smithi008 ceph-mon[66415]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:55:27.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:27 smithi029 ceph-mon[53210]: pgmap v194: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 350 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.9 MiB/s wr, 357 op/s; 261/1623 objects degraded (16.081%) 2024-02-10T06:55:27.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:27 smithi029 ceph-mon[53210]: from='osd.4 [v2:172.21.15.29:6800/3889378905,v1:172.21.15.29:6801/3889378905]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:55:27.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:27 smithi029 ceph-mon[53210]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:55:28.191 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 06:55:27 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T06:55:27.934+0000 7fa555c40700 -1 osd.4 108 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:55:28.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 273188 9.84 MB/sec execute 529 sec latency 56.540 ms 2024-02-10T06:55:28.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:28 smithi008 ceph-mon[66415]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T06:55:28.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:28 smithi008 ceph-mon[66415]: osdmap e111: 8 total, 7 up, 8 in 2024-02-10T06:55:28.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:28 smithi008 ceph-mon[66415]: from='osd.4 [v2:172.21.15.29:6800/3889378905,v1:172.21.15.29:6801/3889378905]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:55:28.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:28 smithi008 ceph-mon[66415]: failed to encode map e111 with expected crc 2024-02-10T06:55:28.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:28 smithi008 ceph-mon[66415]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:55:28.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:28 smithi008 ceph-mon[66415]: failed to encode map e111 with expected crc 2024-02-10T06:55:28.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:28 smithi008 ceph-mon[66415]: from='osd.4 ' entity='osd.4' 2024-02-10T06:55:28.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:28 smithi029 ceph-mon[53210]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T06:55:28.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:28 smithi029 ceph-mon[53210]: osdmap e111: 8 total, 7 up, 8 in 2024-02-10T06:55:28.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:28 smithi029 ceph-mon[53210]: from='osd.4 [v2:172.21.15.29:6800/3889378905,v1:172.21.15.29:6801/3889378905]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:55:28.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:28 smithi029 ceph-mon[53210]: failed to encode map e111 with expected crc 2024-02-10T06:55:28.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:28 smithi029 ceph-mon[53210]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:55:28.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:28 smithi029 ceph-mon[53210]: failed to encode map e111 with expected crc 2024-02-10T06:55:28.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:28 smithi029 ceph-mon[53210]: from='osd.4 ' entity='osd.4' 2024-02-10T06:55:29.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 273284 9.82 MB/sec execute 530 sec latency 704.767 ms 2024-02-10T06:55:30.021 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:29 smithi008 ceph-mon[66415]: pgmap v196: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 353 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.9 MiB/s wr, 532 op/s; 259/1602 objects degraded (16.167%) 2024-02-10T06:55:30.021 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:29 smithi008 ceph-mon[66415]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:55:30.021 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:29 smithi008 ceph-mon[66415]: osd.4 [v2:172.21.15.29:6800/3889378905,v1:172.21.15.29:6801/3889378905] boot 2024-02-10T06:55:30.021 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:29 smithi008 ceph-mon[66415]: osdmap e112: 8 total, 8 up, 8 in 2024-02-10T06:55:30.021 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:29 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:55:30.021 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:29 smithi008 ceph-mon[66415]: failed to encode map e112 with expected crc 2024-02-10T06:55:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:29 smithi029 ceph-mon[53210]: pgmap v196: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 353 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.9 MiB/s wr, 532 op/s; 259/1602 objects degraded (16.167%) 2024-02-10T06:55:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:29 smithi029 ceph-mon[53210]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:55:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:29 smithi029 ceph-mon[53210]: osd.4 [v2:172.21.15.29:6800/3889378905,v1:172.21.15.29:6801/3889378905] boot 2024-02-10T06:55:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:29 smithi029 ceph-mon[53210]: osdmap e112: 8 total, 8 up, 8 in 2024-02-10T06:55:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:29 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:55:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:29 smithi029 ceph-mon[53210]: failed to encode map e112 with expected crc 2024-02-10T06:55:30.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 273376 9.81 MB/sec execute 531 sec latency 1681.046 ms 2024-02-10T06:55:30.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:30 smithi029 ceph-mon[53210]: failed to encode map e112 with expected crc 2024-02-10T06:55:30.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:30 smithi029 ceph-mon[53210]: failed to encode map e112 with expected crc 2024-02-10T06:55:30.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:30 smithi029 ceph-mon[53210]: failed to encode map e112 with expected crc 2024-02-10T06:55:30.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:30 smithi029 ceph-mon[53210]: failed to encode map e112 with expected crc 2024-02-10T06:55:30.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:30 smithi029 ceph-mon[53210]: failed to encode map e112 with expected crc 2024-02-10T06:55:30.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:30 smithi029 ceph-mon[53210]: osdmap e113: 8 total, 8 up, 8 in 2024-02-10T06:55:30.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:30 smithi029 ceph-mon[53210]: failed to encode map e113 with expected crc 2024-02-10T06:55:30.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:30 smithi029 ceph-mon[53210]: failed to encode map e113 with expected crc 2024-02-10T06:55:30.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:30 smithi029 ceph-mon[53210]: failed to encode map e113 with expected crc 2024-02-10T06:55:31.020 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:30 smithi008 ceph-mon[66415]: failed to encode map e112 with expected crc 2024-02-10T06:55:31.020 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:30 smithi008 ceph-mon[66415]: failed to encode map e112 with expected crc 2024-02-10T06:55:31.020 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:30 smithi008 ceph-mon[66415]: failed to encode map e112 with expected crc 2024-02-10T06:55:31.020 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:30 smithi008 ceph-mon[66415]: failed to encode map e112 with expected crc 2024-02-10T06:55:31.020 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:30 smithi008 ceph-mon[66415]: failed to encode map e112 with expected crc 2024-02-10T06:55:31.020 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:30 smithi008 ceph-mon[66415]: osdmap e113: 8 total, 8 up, 8 in 2024-02-10T06:55:31.020 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:30 smithi008 ceph-mon[66415]: failed to encode map e113 with expected crc 2024-02-10T06:55:31.021 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:30 smithi008 ceph-mon[66415]: failed to encode map e113 with expected crc 2024-02-10T06:55:31.021 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:30 smithi008 ceph-mon[66415]: failed to encode map e113 with expected crc 2024-02-10T06:55:31.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 273607 9.80 MB/sec execute 532 sec latency 2009.941 ms 2024-02-10T06:55:31.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: failed to encode map e113 with expected crc 2024-02-10T06:55:31.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: failed to encode map e113 with expected crc 2024-02-10T06:55:31.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: failed to encode map e113 with expected crc 2024-02-10T06:55:31.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: failed to encode map e113 with expected crc 2024-02-10T06:55:31.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: failed to encode map e113 with expected crc 2024-02-10T06:55:31.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: failed to encode map e113 with expected crc 2024-02-10T06:55:31.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: failed to encode map e113 with expected crc 2024-02-10T06:55:31.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: failed to encode map e113 with expected crc 2024-02-10T06:55:31.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: pgmap v199: 97 pgs: 20 peering, 12 active+undersized, 9 active+undersized+degraded, 56 active+clean; 352 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.9 MiB/s wr, 448 op/s; 87/1587 objects degraded (5.482%) 2024-02-10T06:55:31.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:31.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:31.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:55:31.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:55:31.803 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:31.804 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:31.804 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:55:31.804 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:55:31.804 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:55:31.804 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:55:31.804 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:55:31.804 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:55:31.804 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-02-10T06:55:31.804 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: failed to encode map e114 with expected crc 2024-02-10T06:55:31.804 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:31 smithi008 ceph-mon[66415]: osdmap e114: 8 total, 8 up, 8 in 2024-02-10T06:55:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: failed to encode map e113 with expected crc 2024-02-10T06:55:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: failed to encode map e113 with expected crc 2024-02-10T06:55:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: failed to encode map e113 with expected crc 2024-02-10T06:55:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: failed to encode map e113 with expected crc 2024-02-10T06:55:32.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: failed to encode map e113 with expected crc 2024-02-10T06:55:32.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: failed to encode map e113 with expected crc 2024-02-10T06:55:32.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: failed to encode map e113 with expected crc 2024-02-10T06:55:32.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: failed to encode map e113 with expected crc 2024-02-10T06:55:32.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: pgmap v199: 97 pgs: 20 peering, 12 active+undersized, 9 active+undersized+degraded, 56 active+clean; 352 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.9 MiB/s wr, 448 op/s; 87/1587 objects degraded (5.482%) 2024-02-10T06:55:32.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:32.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:32.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:55:32.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:55:32.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:32.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:32.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:55:32.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:55:32.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:55:32.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:55:32.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:55:32.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:55:32.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-02-10T06:55:32.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: failed to encode map e114 with expected crc 2024-02-10T06:55:32.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:31 smithi029 ceph-mon[53210]: osdmap e114: 8 total, 8 up, 8 in 2024-02-10T06:55:32.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 273930 9.81 MB/sec execute 533 sec latency 519.028 ms 2024-02-10T06:55:32.809 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:32 smithi008 ceph-mon[66415]: failed to encode map e114 with expected crc 2024-02-10T06:55:32.809 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:32 smithi008 ceph-mon[66415]: failed to encode map e114 with expected crc 2024-02-10T06:55:32.809 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:32 smithi008 ceph-mon[66415]: failed to encode map e114 with expected crc 2024-02-10T06:55:32.809 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:32 smithi008 ceph-mon[66415]: failed to encode map e115 with expected crc 2024-02-10T06:55:32.809 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:32 smithi008 ceph-mon[66415]: osdmap e115: 8 total, 8 up, 8 in 2024-02-10T06:55:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:32 smithi029 ceph-mon[53210]: failed to encode map e114 with expected crc 2024-02-10T06:55:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:32 smithi029 ceph-mon[53210]: failed to encode map e114 with expected crc 2024-02-10T06:55:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:32 smithi029 ceph-mon[53210]: failed to encode map e114 with expected crc 2024-02-10T06:55:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:32 smithi029 ceph-mon[53210]: failed to encode map e115 with expected crc 2024-02-10T06:55:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:32 smithi029 ceph-mon[53210]: osdmap e115: 8 total, 8 up, 8 in 2024-02-10T06:55:33.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 274212 9.81 MB/sec execute 534 sec latency 838.179 ms 2024-02-10T06:55:33.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:33 smithi008 ceph-mon[66415]: failed to encode map e115 with expected crc 2024-02-10T06:55:33.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:33 smithi008 ceph-mon[66415]: failed to encode map e115 with expected crc 2024-02-10T06:55:33.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:33 smithi008 ceph-mon[66415]: pgmap v202: 97 pgs: 20 peering, 11 active+undersized, 8 active+undersized+degraded, 58 active+clean; 348 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.1 MiB/s wr, 354 op/s; 68/1581 objects degraded (4.301%); 73 KiB/s, 4 objects/s recovering 2024-02-10T06:55:33.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:33 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 68/1581 objects degraded (4.301%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T06:55:33.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:33 smithi008 ceph-mon[66415]: failed to encode map e116 with expected crc 2024-02-10T06:55:33.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:33 smithi008 ceph-mon[66415]: osdmap e116: 8 total, 8 up, 8 in 2024-02-10T06:55:33.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:33 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:33.830 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:33 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:55:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:33 smithi029 ceph-mon[53210]: failed to encode map e115 with expected crc 2024-02-10T06:55:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:33 smithi029 ceph-mon[53210]: failed to encode map e115 with expected crc 2024-02-10T06:55:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:33 smithi029 ceph-mon[53210]: pgmap v202: 97 pgs: 20 peering, 11 active+undersized, 8 active+undersized+degraded, 58 active+clean; 348 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.1 MiB/s wr, 354 op/s; 68/1581 objects degraded (4.301%); 73 KiB/s, 4 objects/s recovering 2024-02-10T06:55:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:33 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 68/1581 objects degraded (4.301%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T06:55:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:33 smithi029 ceph-mon[53210]: failed to encode map e116 with expected crc 2024-02-10T06:55:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:33 smithi029 ceph-mon[53210]: osdmap e116: 8 total, 8 up, 8 in 2024-02-10T06:55:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:33 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:33 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:55:34.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 274587 9.81 MB/sec execute 535 sec latency 1012.340 ms 2024-02-10T06:55:34.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:34 smithi008 ceph-mon[66415]: failed to encode map e116 with expected crc 2024-02-10T06:55:34.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:34 smithi008 ceph-mon[66415]: failed to encode map e116 with expected crc 2024-02-10T06:55:34.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:34 smithi008 ceph-mon[66415]: failed to encode map e116 with expected crc 2024-02-10T06:55:34.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:34 smithi008 ceph-mon[66415]: failed to encode map e116 with expected crc 2024-02-10T06:55:34.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:34 smithi008 ceph-mon[66415]: failed to encode map e116 with expected crc 2024-02-10T06:55:34.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:34 smithi008 ceph-mon[66415]: failed to encode map e116 with expected crc 2024-02-10T06:55:34.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:34 smithi008 ceph-mon[66415]: failed to encode map e116 with expected crc 2024-02-10T06:55:34.846 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:34 smithi008 ceph-mon[66415]: osdmap e117: 8 total, 8 up, 8 in 2024-02-10T06:55:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:34 smithi029 ceph-mon[53210]: failed to encode map e116 with expected crc 2024-02-10T06:55:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:34 smithi029 ceph-mon[53210]: failed to encode map e116 with expected crc 2024-02-10T06:55:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:34 smithi029 ceph-mon[53210]: failed to encode map e116 with expected crc 2024-02-10T06:55:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:34 smithi029 ceph-mon[53210]: failed to encode map e116 with expected crc 2024-02-10T06:55:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:34 smithi029 ceph-mon[53210]: failed to encode map e116 with expected crc 2024-02-10T06:55:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:34 smithi029 ceph-mon[53210]: failed to encode map e116 with expected crc 2024-02-10T06:55:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:34 smithi029 ceph-mon[53210]: failed to encode map e116 with expected crc 2024-02-10T06:55:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:34 smithi029 ceph-mon[53210]: osdmap e117: 8 total, 8 up, 8 in 2024-02-10T06:55:35.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 274879 9.80 MB/sec execute 536 sec latency 38.433 ms 2024-02-10T06:55:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:35 smithi008 ceph-mon[66415]: failed to encode map e117 with expected crc 2024-02-10T06:55:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:35 smithi008 ceph-mon[66415]: failed to encode map e117 with expected crc 2024-02-10T06:55:35.872 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:35 smithi008 ceph-mon[66415]: failed to encode map e117 with expected crc 2024-02-10T06:55:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:35 smithi008 ceph-mon[66415]: pgmap v205: 97 pgs: 7 peering, 90 active+clean; 351 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.5 MiB/s wr, 266 op/s; 426 KiB/s, 63 keys/s, 61 objects/s recovering 2024-02-10T06:55:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:35 smithi008 ceph-mon[66415]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 68/1581 objects degraded (4.301%), 8 pgs degraded) 2024-02-10T06:55:35.873 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:35 smithi008 ceph-mon[66415]: Cluster is now healthy 2024-02-10T06:55:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:35 smithi029 ceph-mon[53210]: failed to encode map e117 with expected crc 2024-02-10T06:55:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:35 smithi029 ceph-mon[53210]: failed to encode map e117 with expected crc 2024-02-10T06:55:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:35 smithi029 ceph-mon[53210]: failed to encode map e117 with expected crc 2024-02-10T06:55:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:35 smithi029 ceph-mon[53210]: pgmap v205: 97 pgs: 7 peering, 90 active+clean; 351 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.5 MiB/s wr, 266 op/s; 426 KiB/s, 63 keys/s, 61 objects/s recovering 2024-02-10T06:55:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:35 smithi029 ceph-mon[53210]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 68/1581 objects degraded (4.301%), 8 pgs degraded) 2024-02-10T06:55:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:35 smithi029 ceph-mon[53210]: Cluster is now healthy 2024-02-10T06:55:36.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 275208 9.80 MB/sec execute 537 sec latency 42.741 ms 2024-02-10T06:55:37.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 275527 9.79 MB/sec execute 538 sec latency 45.507 ms 2024-02-10T06:55:37.922 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:37 smithi008 ceph-mon[66415]: pgmap v206: 97 pgs: 1 peering, 96 active+clean; 352 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.1 MiB/s wr, 216 op/s; 489 KiB/s, 47 keys/s, 56 objects/s recovering 2024-02-10T06:55:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:37 smithi029 ceph-mon[53210]: pgmap v206: 97 pgs: 1 peering, 96 active+clean; 352 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.1 MiB/s wr, 216 op/s; 489 KiB/s, 47 keys/s, 56 objects/s recovering 2024-02-10T06:55:38.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 275896 9.80 MB/sec execute 539 sec latency 41.087 ms 2024-02-10T06:55:39.671 INFO:teuthology.orchestra.run.smithi008.stdout: 5 276276 9.79 MB/sec execute 540 sec latency 35.486 ms 2024-02-10T06:55:39.974 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:39 smithi008 ceph-mon[66415]: pgmap v207: 97 pgs: 97 active+clean; 358 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 4.2 MiB/s wr, 411 op/s; 367 KiB/s, 39 keys/s, 44 objects/s recovering 2024-02-10T06:55:40.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:39 smithi029 ceph-mon[53210]: pgmap v207: 97 pgs: 97 active+clean; 358 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 4.2 MiB/s wr, 411 op/s; 367 KiB/s, 39 keys/s, 44 objects/s recovering 2024-02-10T06:55:40.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 276666 9.79 MB/sec execute 541 sec latency 41.857 ms 2024-02-10T06:55:41.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 277034 9.79 MB/sec execute 542 sec latency 46.875 ms 2024-02-10T06:55:42.019 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:41 smithi008 ceph-mon[66415]: pgmap v208: 97 pgs: 97 active+clean; 345 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.7 MiB/s wr, 412 op/s; 292 KiB/s, 31 keys/s, 35 objects/s recovering 2024-02-10T06:55:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:41 smithi029 ceph-mon[53210]: pgmap v208: 97 pgs: 97 active+clean; 345 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.7 MiB/s wr, 412 op/s; 292 KiB/s, 31 keys/s, 35 objects/s recovering 2024-02-10T06:55:42.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 277367 9.79 MB/sec execute 543 sec latency 41.350 ms 2024-02-10T06:55:43.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 277726 9.81 MB/sec execute 544 sec latency 46.719 ms 2024-02-10T06:55:43.822 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:43 smithi008 ceph-mon[66415]: pgmap v209: 97 pgs: 97 active+clean; 351 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.9 MiB/s wr, 535 op/s; 101 KiB/s, 27 keys/s, 24 objects/s recovering 2024-02-10T06:55:43.822 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:43 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:44.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:43 smithi029 ceph-mon[53210]: pgmap v209: 97 pgs: 97 active+clean; 351 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.9 MiB/s wr, 535 op/s; 101 KiB/s, 27 keys/s, 24 objects/s recovering 2024-02-10T06:55:44.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:43 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:44.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 278070 9.80 MB/sec execute 545 sec latency 48.330 ms 2024-02-10T06:55:45.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 278318 9.79 MB/sec execute 546 sec latency 39.333 ms 2024-02-10T06:55:45.839 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:45 smithi008 ceph-mon[66415]: pgmap v210: 97 pgs: 97 active+clean; 345 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.4 MiB/s wr, 537 op/s; 89 KiB/s, 19 objects/s recovering 2024-02-10T06:55:46.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:45 smithi029 ceph-mon[53210]: pgmap v210: 97 pgs: 97 active+clean; 345 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.4 MiB/s wr, 537 op/s; 89 KiB/s, 19 objects/s recovering 2024-02-10T06:55:46.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 278621 9.79 MB/sec execute 547 sec latency 52.619 ms 2024-02-10T06:55:46.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:46 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:55:46.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:46 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:55:46.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:46 smithi008 ceph-mon[66415]: Upgrade: osd.5 is safe to restart 2024-02-10T06:55:46.892 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:46 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:55:46.892 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:46 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T06:55:46.892 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:46 smithi029 ceph-mon[53210]: Upgrade: osd.5 is safe to restart 2024-02-10T06:55:47.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 278854 9.78 MB/sec execute 548 sec latency 77.822 ms 2024-02-10T06:55:47.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:47 smithi008 ceph-mon[66415]: pgmap v211: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.3 MiB/s wr, 418 op/s; 77 KiB/s, 4 objects/s recovering 2024-02-10T06:55:47.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:47 smithi029 ceph-mon[53210]: pgmap v211: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.3 MiB/s wr, 418 op/s; 77 KiB/s, 4 objects/s recovering 2024-02-10T06:55:48.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 279175 9.77 MB/sec execute 549 sec latency 45.606 ms 2024-02-10T06:55:49.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:49 smithi008 ceph-mon[66415]: pgmap v212: 97 pgs: 97 active+clean; 353 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.2 MiB/s wr, 560 op/s 2024-02-10T06:55:49.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:49.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:55:49.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:49 smithi008 ceph-mon[66415]: Upgrade: Updating osd.5 2024-02-10T06:55:49.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:49.480 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T06:55:49.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:55:49.481 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:49 smithi008 ceph-mon[66415]: Deploying daemon osd.5 on smithi029 2024-02-10T06:55:49.583 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:49 smithi029 ceph-mon[53210]: pgmap v212: 97 pgs: 97 active+clean; 353 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.2 MiB/s wr, 560 op/s 2024-02-10T06:55:49.583 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:49.583 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:55:49.583 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:49 smithi029 ceph-mon[53210]: Upgrade: Updating osd.5 2024-02-10T06:55:49.583 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:49.584 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T06:55:49.584 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:55:49.584 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:49 smithi029 ceph-mon[53210]: Deploying daemon osd.5 on smithi029 2024-02-10T06:55:49.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 279484 9.78 MB/sec execute 550 sec latency 71.676 ms 2024-02-10T06:55:50.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 279841 9.77 MB/sec execute 551 sec latency 79.142 ms 2024-02-10T06:55:51.517 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:51 smithi008 ceph-mon[66415]: pgmap v213: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 486 op/s 2024-02-10T06:55:51.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 280194 9.77 MB/sec execute 552 sec latency 47.643 ms 2024-02-10T06:55:51.699 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:51 smithi029 ceph-mon[53210]: pgmap v213: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 486 op/s 2024-02-10T06:55:51.954 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:55:52.550 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:52 smithi008 ceph-mon[66415]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:55:52.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 280464 9.76 MB/sec execute 553 sec latency 45.033 ms 2024-02-10T06:55:52.708 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:52 smithi029 systemd[1]: Stopping Ceph osd.5 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:55:52.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:52 smithi029 ceph-mon[53210]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:55:52.965 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:55:52.965 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (8m) 54s ago 17m 15.6M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:55:52.965 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (4m) 54s ago 17m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T06:55:52.965 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (4m) 29s ago 17m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T06:55:52.965 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (7m) 54s ago 17m 43.0M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:55:52.965 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (11m) 54s ago 11m 38.5M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:55:52.965 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (11m) 29s ago 11m 15.4M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:55:52.965 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (7m) 54s ago 19m 507M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:55:52.966 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (6m) 29s ago 17m 451M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:55:52.966 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (5m) 54s ago 19m 53.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:55:52.966 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (5m) 29s ago 17m 41.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:55:52.966 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (11m) 54s ago 11m 271M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:55:52.966 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (11m) 29s ago 11m 84.6M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:55:52.966 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (7m) 54s ago 17m 11.4M - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:55:52.966 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (7m) 29s ago 16m 17.2M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:55:52.966 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (3m) 54s ago 16m 355M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 acb0107eae6c 2024-02-10T06:55:52.966 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (2m) 54s ago 15m 282M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 09363f1c35ae 2024-02-10T06:55:52.966 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (113s) 54s ago 15m 237M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 bd6bbb022085 2024-02-10T06:55:52.966 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (64s) 54s ago 15m 15.0M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc3683d167e9 2024-02-10T06:55:52.966 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (34s) 29s ago 14m 14.8M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 2b1f7cbc27ce 2024-02-10T06:55:52.966 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (14m) 29s ago 14m 649M 2705M 17.2.0 e1d6a67b021e bb09e35d5f0d 2024-02-10T06:55:52.967 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (13m) 29s ago 13m 566M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:55:52.967 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (13m) 29s ago 13m 588M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:55:52.967 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (6m) 54s ago 16m 63.3M - 2.33.4 514e6a882f6e 193819b86dea 2024-02-10T06:55:53.208 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:52 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[34293]: 2024-02-10T06:55:52.772+0000 7f8ba05b2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:55:53.208 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:52 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[34293]: 2024-02-10T06:55:52.772+0000 7f8ba05b2700 -1 osd.5 117 *** Got signal Terminated *** 2024-02-10T06:55:53.208 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:52 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[34293]: 2024-02-10T06:55:52.772+0000 7f8ba05b2700 -1 osd.5 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:55:53.589 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:53 smithi008 ceph-mon[66415]: pgmap v214: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.6 MiB/s wr, 490 op/s 2024-02-10T06:55:53.589 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:53 smithi008 ceph-mon[66415]: from='client.34390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:55:53.589 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:53 smithi008 ceph-mon[66415]: osd.5 marked itself down and dead 2024-02-10T06:55:53.589 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:53 smithi008 ceph-mon[66415]: from='client.34394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:55:53.589 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:55:53.590 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:55:53.590 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:55:53.590 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:55:53.590 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:55:53.590 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:55:53.590 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:55:53.590 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:55:53.590 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-10T06:55:53.590 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 5 2024-02-10T06:55:53.590 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:55:53.590 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:55:53.590 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:55:53.591 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:55:53.591 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:55:53.591 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-10T06:55:53.591 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 9 2024-02-10T06:55:53.591 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:55:53.591 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:55:53.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 280809 9.77 MB/sec execute 554 sec latency 252.336 ms 2024-02-10T06:55:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:53 smithi029 ceph-mon[53210]: pgmap v214: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.6 MiB/s wr, 490 op/s 2024-02-10T06:55:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:53 smithi029 ceph-mon[53210]: from='client.34390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:55:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:53 smithi029 ceph-mon[53210]: osd.5 marked itself down and dead 2024-02-10T06:55:53.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:53 smithi029 ceph-mon[53210]: from='client.34394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:55:53.934 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:55:53.935 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:55:53.935 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:55:53.935 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:55:53.935 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:55:53.935 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:55:53.935 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-02-10T06:55:53.935 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-02-10T06:55:53.935 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:55:53.935 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "11/23 daemons upgraded", 2024-02-10T06:55:53.935 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:55:53.935 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:55:53.935 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:55:54.332 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:54 smithi029 podman[59938]: 2024-02-10 06:55:54.038144754 +0000 UTC m=+1.398077855 container died bb09e35d5f0dfa1fc6bac5b70c844553239fb5f08b19c129b6c88ba7ddeb0fa3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, version=8, GIT_CLEAN=True, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-10T06:55:54.342 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 osds down 2024-02-10T06:55:54.342 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-10T06:55:54.342 INFO:teuthology.orchestra.run.smithi008.stdout: osd.5 (root=default,host=smithi029) is down 2024-02-10T06:55:54.594 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:54 smithi008 ceph-mon[66415]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:55:54.594 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:54 smithi008 ceph-mon[66415]: failed to encode map e118 with expected crc 2024-02-10T06:55:54.594 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:54 smithi008 ceph-mon[66415]: failed to encode map e118 with expected crc 2024-02-10T06:55:54.594 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:54 smithi008 ceph-mon[66415]: osdmap e118: 8 total, 7 up, 8 in 2024-02-10T06:55:54.594 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:54 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/1153478985' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:55:54.595 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:54 smithi008 ceph-mon[66415]: from='client.34402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:55:54.595 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:54 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/1194088382' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:55:54.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 280908 9.76 MB/sec execute 555 sec latency 1002.551 ms 2024-02-10T06:55:54.708 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:54 smithi029 podman[59938]: 2024-02-10 06:55:54.432121186 +0000 UTC m=+1.792054286 container cleanup bb09e35d5f0dfa1fc6bac5b70c844553239fb5f08b19c129b6c88ba7ddeb0fa3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, com.redhat.component=centos-stream-container, name=centos-stream, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, version=8, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., RELEASE=HEAD, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-10T06:55:54.708 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:54 smithi029 bash[59938]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5 2024-02-10T06:55:54.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:54 smithi029 ceph-mon[53210]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:55:54.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:54 smithi029 ceph-mon[53210]: failed to encode map e118 with expected crc 2024-02-10T06:55:54.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:54 smithi029 ceph-mon[53210]: failed to encode map e118 with expected crc 2024-02-10T06:55:54.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:54 smithi029 ceph-mon[53210]: osdmap e118: 8 total, 7 up, 8 in 2024-02-10T06:55:54.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:54 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/1153478985' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:55:54.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:54 smithi029 ceph-mon[53210]: from='client.34402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:55:54.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:54 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/1194088382' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:55:55.379 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:55 smithi029 podman[59954]: 2024-02-10 06:55:55.087741347 +0000 UTC m=+1.050617350 container remove bb09e35d5f0dfa1fc6bac5b70c844553239fb5f08b19c129b6c88ba7ddeb0fa3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_CLEAN=True, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.openshift.expose-services=, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.buildah.version=1.19.8, release=754, GIT_BRANCH=HEAD) 2024-02-10T06:55:55.482 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:55 smithi008 ceph-mon[66415]: pgmap v216: 97 pgs: 15 peering, 7 stale+active+clean, 75 active+clean; 346 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 493 op/s 2024-02-10T06:55:55.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:55 smithi008 ceph-mon[66415]: osdmap e119: 8 total, 7 up, 8 in 2024-02-10T06:55:55.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:55 smithi008 ceph-mon[66415]: failed to encode map e119 with expected crc 2024-02-10T06:55:55.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:55 smithi008 ceph-mon[66415]: failed to encode map e119 with expected crc 2024-02-10T06:55:55.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:55 smithi008 ceph-mon[66415]: failed to encode map e119 with expected crc 2024-02-10T06:55:55.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:55 smithi008 ceph-mon[66415]: failed to encode map e119 with expected crc 2024-02-10T06:55:55.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:55 smithi008 ceph-mon[66415]: failed to encode map e119 with expected crc 2024-02-10T06:55:55.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:55 smithi008 ceph-mon[66415]: failed to encode map e119 with expected crc 2024-02-10T06:55:55.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:55 smithi008 ceph-mon[66415]: failed to encode map e119 with expected crc 2024-02-10T06:55:55.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:55 smithi008 ceph-mon[66415]: failed to encode map e119 with expected crc 2024-02-10T06:55:55.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:55 smithi008 ceph-mon[66415]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-02-10T06:55:55.630 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:55 smithi029 podman[60042]: 2024-02-10 06:55:55.367584437 +0000 UTC m=+0.020038848 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:55:55.630 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:55 smithi029 podman[60042]: 2024-02-10 06:55:55.481814192 +0000 UTC m=+0.134268564 container create 0fe8d701324f71ad93f578ac2ba0465ed58c9275bdc5eeca3df851b556f07138 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-10T06:55:55.630 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:55 smithi029 ceph-mon[53210]: pgmap v216: 97 pgs: 15 peering, 7 stale+active+clean, 75 active+clean; 346 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 493 op/s 2024-02-10T06:55:55.630 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:55 smithi029 ceph-mon[53210]: osdmap e119: 8 total, 7 up, 8 in 2024-02-10T06:55:55.630 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:55 smithi029 ceph-mon[53210]: failed to encode map e119 with expected crc 2024-02-10T06:55:55.631 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:55 smithi029 ceph-mon[53210]: failed to encode map e119 with expected crc 2024-02-10T06:55:55.631 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:55 smithi029 ceph-mon[53210]: failed to encode map e119 with expected crc 2024-02-10T06:55:55.631 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:55 smithi029 ceph-mon[53210]: failed to encode map e119 with expected crc 2024-02-10T06:55:55.631 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:55 smithi029 ceph-mon[53210]: failed to encode map e119 with expected crc 2024-02-10T06:55:55.631 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:55 smithi029 ceph-mon[53210]: failed to encode map e119 with expected crc 2024-02-10T06:55:55.631 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:55 smithi029 ceph-mon[53210]: failed to encode map e119 with expected crc 2024-02-10T06:55:55.631 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:55 smithi029 ceph-mon[53210]: failed to encode map e119 with expected crc 2024-02-10T06:55:55.631 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:55 smithi029 ceph-mon[53210]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-02-10T06:55:55.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 281320 9.77 MB/sec execute 556 sec latency 61.804 ms 2024-02-10T06:55:55.958 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:55 smithi029 podman[60042]: 2024-02-10 06:55:55.764220634 +0000 UTC m=+0.416675025 container init 0fe8d701324f71ad93f578ac2ba0465ed58c9275bdc5eeca3df851b556f07138 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, org.label-schema.schema-version=1.0) 2024-02-10T06:55:55.958 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:55 smithi029 podman[60042]: 2024-02-10 06:55:55.768635931 +0000 UTC m=+0.421090312 container start 0fe8d701324f71ad93f578ac2ba0465ed58c9275bdc5eeca3df851b556f07138 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-10T06:55:55.958 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:55 smithi029 podman[60042]: 2024-02-10 06:55:55.873226799 +0000 UTC m=+0.525681179 container attach 0fe8d701324f71ad93f578ac2ba0465ed58c9275bdc5eeca3df851b556f07138 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-deactivate, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, ceph=True) 2024-02-10T06:55:56.433 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:56 smithi029 podman[60042]: 2024-02-10 06:55:56.117544174 +0000 UTC m=+0.769998557 container died 0fe8d701324f71ad93f578ac2ba0465ed58c9275bdc5eeca3df851b556f07138 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-10T06:55:56.490 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:56 smithi008 ceph-mon[66415]: Health check failed: Degraded data redundancy: 18/1665 objects degraded (1.081%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T06:55:56.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 281697 9.77 MB/sec execute 557 sec latency 42.236 ms 2024-02-10T06:55:56.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:56 smithi029 ceph-mon[53210]: Health check failed: Degraded data redundancy: 18/1665 objects degraded (1.081%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T06:55:57.100 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:56 smithi029 podman[60067]: 2024-02-10 06:55:56.834236237 +0000 UTC m=+0.711164492 container remove 0fe8d701324f71ad93f578ac2ba0465ed58c9275bdc5eeca3df851b556f07138 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=main-290212a, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux ) 2024-02-10T06:55:57.100 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:56 smithi029 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.5.service: Deactivated successfully. 2024-02-10T06:55:57.100 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:56 smithi029 systemd[1]: Stopped Ceph osd.5 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:55:57.100 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:56 smithi029 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.5.service: Consumed 5min 49.948s CPU time. 2024-02-10T06:55:57.370 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:57 smithi029 systemd[1]: Starting Ceph osd.5 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:55:57.370 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:57 smithi029 podman[60169]: 2024-02-10 06:55:57.249841362 +0000 UTC m=+0.022359705 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:55:57.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:57 smithi008 ceph-mon[66415]: pgmap v218: 97 pgs: 1 active+undersized, 2 active+undersized+degraded, 15 peering, 7 stale+active+clean, 72 active+clean; 349 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.5 MiB/s wr, 435 op/s; 18/1665 objects degraded (1.081%) 2024-02-10T06:55:57.641 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:57 smithi029 ceph-mon[53210]: pgmap v218: 97 pgs: 1 active+undersized, 2 active+undersized+degraded, 15 peering, 7 stale+active+clean, 72 active+clean; 349 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.5 MiB/s wr, 435 op/s; 18/1665 objects degraded (1.081%) 2024-02-10T06:55:57.641 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:57 smithi029 podman[60169]: 2024-02-10 06:55:57.36965216 +0000 UTC m=+0.142170481 container create 3c57d741dea3f250acab93cc5c31d5fd5bb7fee6b601a126587a59091dcde924 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-activate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:55:57.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 282008 9.76 MB/sec execute 558 sec latency 44.598 ms 2024-02-10T06:55:57.958 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:57 smithi029 podman[60169]: 2024-02-10 06:55:57.640591772 +0000 UTC m=+0.413110108 container init 3c57d741dea3f250acab93cc5c31d5fd5bb7fee6b601a126587a59091dcde924 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, ceph=True) 2024-02-10T06:55:57.958 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:57 smithi029 podman[60169]: 2024-02-10 06:55:57.646122459 +0000 UTC m=+0.418640785 container start 3c57d741dea3f250acab93cc5c31d5fd5bb7fee6b601a126587a59091dcde924 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-10T06:55:57.958 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:57 smithi029 podman[60169]: 2024-02-10 06:55:57.749630498 +0000 UTC m=+0.522148826 container attach 3c57d741dea3f250acab93cc5c31d5fd5bb7fee6b601a126587a59091dcde924 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-activate, RELEASE=main-290212a, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-02-10T06:55:58.582 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:58 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-activate[60183]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T06:55:58.583 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:58 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-activate[60183]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T06:55:58.583 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:58 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-activate[60183]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T06:55:58.583 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:58 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-activate[60183]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T06:55:58.583 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:58 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-activate[60183]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-10T06:55:58.583 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:58 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-activate[60183]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T06:55:58.583 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:58 smithi029 bash[60169]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T06:55:58.583 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:58 smithi029 bash[60169]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T06:55:58.583 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:58 smithi029 bash[60169]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T06:55:58.583 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:58 smithi029 bash[60169]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T06:55:58.583 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:58 smithi029 bash[60169]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-10T06:55:58.583 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:58 smithi029 bash[60169]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T06:55:58.583 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:58 smithi029 bash[60169]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-10T06:55:58.584 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:58 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-activate[60183]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-10T06:55:58.584 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:58 smithi029 podman[60169]: 2024-02-10 06:55:58.267556017 +0000 UTC m=+1.040074347 container died 3c57d741dea3f250acab93cc5c31d5fd5bb7fee6b601a126587a59091dcde924 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-activate, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, RELEASE=main-290212a) 2024-02-10T06:55:58.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 282418 9.76 MB/sec execute 559 sec latency 53.252 ms 2024-02-10T06:55:59.172 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:58 smithi029 podman[60310]: 2024-02-10 06:55:58.903814359 +0000 UTC m=+0.630366867 container remove 3c57d741dea3f250acab93cc5c31d5fd5bb7fee6b601a126587a59091dcde924 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-activate, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2) 2024-02-10T06:55:59.172 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:59 smithi029 podman[60366]: 2024-02-10 06:55:59.053062331 +0000 UTC m=+0.022028864 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:55:59.172 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:59 smithi029 podman[60366]: 2024-02-10 06:55:59.172519432 +0000 UTC m=+0.141485961 container create ea4052b4cafaafbfa0cc5018e0d297731b2f70e782a7df7ca641843e67be91ad (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=) 2024-02-10T06:55:59.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 282773 9.76 MB/sec execute 560 sec latency 73.656 ms 2024-02-10T06:55:59.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:59 smithi029 ceph-mon[53210]: pgmap v219: 97 pgs: 4 active+undersized, 12 active+undersized+degraded, 15 peering, 66 active+clean; 349 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.1 KiB/s rd, 4.1 MiB/s wr, 436 op/s; 83/1671 objects degraded (4.967%) 2024-02-10T06:55:59.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:59 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:59.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:59 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:59.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:55:59 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:55:59.708 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:59 smithi029 podman[60366]: 2024-02-10 06:55:59.452802963 +0000 UTC m=+0.421769507 container init ea4052b4cafaafbfa0cc5018e0d297731b2f70e782a7df7ca641843e67be91ad (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:55:59.708 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:59 smithi029 podman[60366]: 2024-02-10 06:55:59.456293731 +0000 UTC m=+0.425260263 container start ea4052b4cafaafbfa0cc5018e0d297731b2f70e782a7df7ca641843e67be91ad (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-290212a) 2024-02-10T06:55:59.708 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:59 smithi029 ceph-osd[60386]: -- 172.21.15.29:0/2308639986 <== mon.1 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558f581e91e0 con 0x558f581bac00 2024-02-10T06:55:59.708 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:59 smithi029 bash[60366]: ea4052b4cafaafbfa0cc5018e0d297731b2f70e782a7df7ca641843e67be91ad 2024-02-10T06:55:59.708 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:59 smithi029 systemd[1]: Started Ceph osd.5 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:55:59.766 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:59 smithi008 ceph-mon[66415]: pgmap v219: 97 pgs: 4 active+undersized, 12 active+undersized+degraded, 15 peering, 66 active+clean; 349 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.1 KiB/s rd, 4.1 MiB/s wr, 436 op/s; 83/1671 objects degraded (4.967%) 2024-02-10T06:55:59.766 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:59 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:59.766 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:59 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:55:59.766 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:55:59 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:56:00.458 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:55:59 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T06:55:59.995+0000 7fb5372007c0 -1 Falling back to public interface 2024-02-10T06:56:00.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 283058 9.76 MB/sec execute 561 sec latency 92.272 ms 2024-02-10T06:56:00.775 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:00 smithi008 ceph-mon[66415]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-02-10T06:56:01.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:00 smithi029 ceph-mon[53210]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-02-10T06:56:01.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 283415 9.75 MB/sec execute 562 sec latency 44.364 ms 2024-02-10T06:56:01.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:01 smithi008 ceph-mon[66415]: pgmap v220: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 349 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.4 MiB/s wr, 502 op/s; 165/1644 objects degraded (10.036%) 2024-02-10T06:56:02.003 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:01 smithi029 ceph-mon[53210]: pgmap v220: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 349 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.4 MiB/s wr, 502 op/s; 165/1644 objects degraded (10.036%) 2024-02-10T06:56:02.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 283773 9.75 MB/sec execute 563 sec latency 42.630 ms 2024-02-10T06:56:03.067 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:02 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 165/1632 objects degraded (10.110%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T06:56:03.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:02 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 165/1632 objects degraded (10.110%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T06:56:03.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 284101 9.74 MB/sec execute 564 sec latency 53.748 ms 2024-02-10T06:56:03.824 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:03 smithi008 ceph-mon[66415]: pgmap v221: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 348 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.8 MiB/s wr, 455 op/s; 165/1632 objects degraded (10.110%) 2024-02-10T06:56:03.824 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:03.824 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:56:04.012 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:03 smithi029 ceph-mon[53210]: pgmap v221: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 348 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.8 MiB/s wr, 455 op/s; 165/1632 objects degraded (10.110%) 2024-02-10T06:56:04.013 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:04.013 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:56:04.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 284413 9.75 MB/sec execute 565 sec latency 39.936 ms 2024-02-10T06:56:05.343 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:05.343 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:05.343 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:05 smithi008 ceph-mon[66415]: pgmap v222: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.9 MiB/s wr, 498 op/s; 165/1581 objects degraded (10.436%) 2024-02-10T06:56:05.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 284777 9.76 MB/sec execute 566 sec latency 68.121 ms 2024-02-10T06:56:05.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:05.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:05.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:05 smithi029 ceph-mon[53210]: pgmap v222: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.9 MiB/s wr, 498 op/s; 165/1581 objects degraded (10.436%) 2024-02-10T06:56:06.192 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:56:05 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T06:56:05.903+0000 7fb5372007c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-02-10T06:56:06.607 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:06 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:06.607 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:06 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:06.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 285078 9.75 MB/sec execute 567 sec latency 49.423 ms 2024-02-10T06:56:06.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:06 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:06.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:06 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:07.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:07 smithi008 ceph-mon[66415]: pgmap v223: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.4 MiB/s wr, 432 op/s; 163/1569 objects degraded (10.389%) 2024-02-10T06:56:07.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:07 smithi008 ceph-mon[66415]: from='osd.5 [v2:172.21.15.29:6808/257240982,v1:172.21.15.29:6809/257240982]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:56:07.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:07 smithi008 ceph-mon[66415]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:56:07.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 285277 9.75 MB/sec execute 568 sec latency 83.191 ms 2024-02-10T06:56:07.839 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:56:07 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T06:56:07.484+0000 7fb5372007c0 -1 osd.5 117 log_to_monitors true 2024-02-10T06:56:07.839 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:07 smithi029 ceph-mon[53210]: pgmap v223: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.4 MiB/s wr, 432 op/s; 163/1569 objects degraded (10.389%) 2024-02-10T06:56:07.839 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:07 smithi029 ceph-mon[53210]: from='osd.5 [v2:172.21.15.29:6808/257240982,v1:172.21.15.29:6809/257240982]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:56:07.839 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:07 smithi029 ceph-mon[53210]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:56:08.208 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 06:56:07 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T06:56:07.843+0000 7fb524650700 -1 osd.5 117 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:56:08.673 INFO:teuthology.orchestra.run.smithi008.stdout: 5 285423 9.73 MB/sec execute 569 sec latency 70.622 ms 2024-02-10T06:56:08.687 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:08 smithi008 ceph-mon[66415]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T06:56:08.687 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:08 smithi008 ceph-mon[66415]: failed to encode map e120 with expected crc 2024-02-10T06:56:08.687 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:08 smithi008 ceph-mon[66415]: osdmap e120: 8 total, 7 up, 8 in 2024-02-10T06:56:08.687 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:08 smithi008 ceph-mon[66415]: failed to encode map e120 with expected crc 2024-02-10T06:56:08.687 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:08 smithi008 ceph-mon[66415]: from='osd.5 [v2:172.21.15.29:6808/257240982,v1:172.21.15.29:6809/257240982]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:56:08.687 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:08 smithi008 ceph-mon[66415]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:56:08.687 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:08 smithi008 ceph-mon[66415]: from='osd.5 ' entity='osd.5' 2024-02-10T06:56:08.854 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:08 smithi029 ceph-mon[53210]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T06:56:08.854 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:08 smithi029 ceph-mon[53210]: failed to encode map e120 with expected crc 2024-02-10T06:56:08.854 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:08 smithi029 ceph-mon[53210]: osdmap e120: 8 total, 7 up, 8 in 2024-02-10T06:56:08.854 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:08 smithi029 ceph-mon[53210]: failed to encode map e120 with expected crc 2024-02-10T06:56:08.854 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:08 smithi029 ceph-mon[53210]: from='osd.5 [v2:172.21.15.29:6808/257240982,v1:172.21.15.29:6809/257240982]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:56:08.855 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:08 smithi029 ceph-mon[53210]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:56:08.855 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:08 smithi029 ceph-mon[53210]: from='osd.5 ' entity='osd.5' 2024-02-10T06:56:09.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 285662 9.73 MB/sec execute 570 sec latency 588.262 ms 2024-02-10T06:56:09.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:09 smithi008 ceph-mon[66415]: pgmap v225: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 352 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.3 MiB/s wr, 540 op/s; 175/1644 objects degraded (10.645%) 2024-02-10T06:56:09.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:09 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 163/1569 objects degraded (10.389%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T06:56:09.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:09 smithi008 ceph-mon[66415]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:56:09.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:09 smithi008 ceph-mon[66415]: osd.5 [v2:172.21.15.29:6808/257240982,v1:172.21.15.29:6809/257240982] boot 2024-02-10T06:56:09.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:09 smithi008 ceph-mon[66415]: osdmap e121: 8 total, 8 up, 8 in 2024-02-10T06:56:09.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:09 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:56:09.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:09 smithi029 ceph-mon[53210]: pgmap v225: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 352 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.3 MiB/s wr, 540 op/s; 175/1644 objects degraded (10.645%) 2024-02-10T06:56:09.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:09 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 163/1569 objects degraded (10.389%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T06:56:09.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:09 smithi029 ceph-mon[53210]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:56:09.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:09 smithi029 ceph-mon[53210]: osd.5 [v2:172.21.15.29:6808/257240982,v1:172.21.15.29:6809/257240982] boot 2024-02-10T06:56:09.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:09 smithi029 ceph-mon[53210]: osdmap e121: 8 total, 8 up, 8 in 2024-02-10T06:56:09.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:09 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:56:10.673 INFO:teuthology.orchestra.run.smithi008.stdout: 5 285859 9.72 MB/sec execute 571 sec latency 824.990 ms 2024-02-10T06:56:10.731 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: failed to encode map e121 with expected crc 2024-02-10T06:56:10.731 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: failed to encode map e121 with expected crc 2024-02-10T06:56:10.731 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: failed to encode map e121 with expected crc 2024-02-10T06:56:10.731 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: failed to encode map e121 with expected crc 2024-02-10T06:56:10.731 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: failed to encode map e121 with expected crc 2024-02-10T06:56:10.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: failed to encode map e121 with expected crc 2024-02-10T06:56:10.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: failed to encode map e121 with expected crc 2024-02-10T06:56:10.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:10.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:10.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:56:10.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:56:10.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:10.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:10.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:56:10.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:56:10.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:56:10.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:56:10.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:56:10.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:56:10.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-10T06:56:10.733 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:10 smithi008 ceph-mon[66415]: osdmap e122: 8 total, 8 up, 8 in 2024-02-10T06:56:10.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: failed to encode map e121 with expected crc 2024-02-10T06:56:10.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: failed to encode map e121 with expected crc 2024-02-10T06:56:10.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: failed to encode map e121 with expected crc 2024-02-10T06:56:10.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: failed to encode map e121 with expected crc 2024-02-10T06:56:10.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: failed to encode map e121 with expected crc 2024-02-10T06:56:10.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: failed to encode map e121 with expected crc 2024-02-10T06:56:10.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: failed to encode map e121 with expected crc 2024-02-10T06:56:10.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:10.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:10.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:56:10.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:56:10.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:10.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:10.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:56:10.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:56:10.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:56:10.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:56:10.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:56:10.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:56:10.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-10T06:56:10.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:10 smithi029 ceph-mon[53210]: osdmap e122: 8 total, 8 up, 8 in 2024-02-10T06:56:11.673 INFO:teuthology.orchestra.run.smithi008.stdout: 5 286051 9.71 MB/sec execute 572 sec latency 127.143 ms 2024-02-10T06:56:11.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:11 smithi008 ceph-mon[66415]: failed to encode map e122 with expected crc 2024-02-10T06:56:11.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:11 smithi008 ceph-mon[66415]: failed to encode map e122 with expected crc 2024-02-10T06:56:11.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:11 smithi008 ceph-mon[66415]: failed to encode map e122 with expected crc 2024-02-10T06:56:11.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:11 smithi008 ceph-mon[66415]: failed to encode map e122 with expected crc 2024-02-10T06:56:11.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:11 smithi008 ceph-mon[66415]: failed to encode map e122 with expected crc 2024-02-10T06:56:11.780 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:11 smithi008 ceph-mon[66415]: failed to encode map e122 with expected crc 2024-02-10T06:56:11.781 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:11 smithi008 ceph-mon[66415]: pgmap v228: 97 pgs: 10 peering, 6 active+undersized, 15 active+undersized+degraded, 66 active+clean; 355 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 4.8 MiB/s wr, 432 op/s; 116/1647 objects degraded (7.043%) 2024-02-10T06:56:11.781 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:11 smithi008 ceph-mon[66415]: osdmap e123: 8 total, 8 up, 8 in 2024-02-10T06:56:11.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:11 smithi029 ceph-mon[53210]: failed to encode map e122 with expected crc 2024-02-10T06:56:11.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:11 smithi029 ceph-mon[53210]: failed to encode map e122 with expected crc 2024-02-10T06:56:11.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:11 smithi029 ceph-mon[53210]: failed to encode map e122 with expected crc 2024-02-10T06:56:11.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:11 smithi029 ceph-mon[53210]: failed to encode map e122 with expected crc 2024-02-10T06:56:11.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:11 smithi029 ceph-mon[53210]: failed to encode map e122 with expected crc 2024-02-10T06:56:11.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:11 smithi029 ceph-mon[53210]: failed to encode map e122 with expected crc 2024-02-10T06:56:11.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:11 smithi029 ceph-mon[53210]: pgmap v228: 97 pgs: 10 peering, 6 active+undersized, 15 active+undersized+degraded, 66 active+clean; 355 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 4.8 MiB/s wr, 432 op/s; 116/1647 objects degraded (7.043%) 2024-02-10T06:56:11.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:11 smithi029 ceph-mon[53210]: osdmap e123: 8 total, 8 up, 8 in 2024-02-10T06:56:12.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 286302 9.70 MB/sec execute 573 sec latency 674.801 ms 2024-02-10T06:56:12.801 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:12 smithi008 ceph-mon[66415]: failed to encode map e123 with expected crc 2024-02-10T06:56:12.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:12 smithi008 ceph-mon[66415]: failed to encode map e123 with expected crc 2024-02-10T06:56:12.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:12 smithi008 ceph-mon[66415]: failed to encode map e123 with expected crc 2024-02-10T06:56:12.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:12 smithi008 ceph-mon[66415]: failed to encode map e123 with expected crc 2024-02-10T06:56:12.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:12 smithi008 ceph-mon[66415]: failed to encode map e123 with expected crc 2024-02-10T06:56:12.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:12 smithi008 ceph-mon[66415]: failed to encode map e123 with expected crc 2024-02-10T06:56:12.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:12 smithi008 ceph-mon[66415]: osdmap e124: 8 total, 8 up, 8 in 2024-02-10T06:56:12.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:12 smithi029 ceph-mon[53210]: failed to encode map e123 with expected crc 2024-02-10T06:56:12.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:12 smithi029 ceph-mon[53210]: failed to encode map e123 with expected crc 2024-02-10T06:56:12.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:12 smithi029 ceph-mon[53210]: failed to encode map e123 with expected crc 2024-02-10T06:56:12.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:12 smithi029 ceph-mon[53210]: failed to encode map e123 with expected crc 2024-02-10T06:56:12.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:12 smithi029 ceph-mon[53210]: failed to encode map e123 with expected crc 2024-02-10T06:56:12.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:12 smithi029 ceph-mon[53210]: failed to encode map e123 with expected crc 2024-02-10T06:56:12.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:12 smithi029 ceph-mon[53210]: osdmap e124: 8 total, 8 up, 8 in 2024-02-10T06:56:13.673 INFO:teuthology.orchestra.run.smithi008.stdout: 5 286539 9.70 MB/sec execute 574 sec latency 881.693 ms 2024-02-10T06:56:13.839 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:13 smithi008 ceph-mon[66415]: failed to encode map e124 with expected crc 2024-02-10T06:56:13.839 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:13 smithi008 ceph-mon[66415]: pgmap v231: 97 pgs: 10 peering, 5 active+undersized, 1 active+undersized+remapped, 13 active+undersized+degraded, 68 active+clean; 348 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.9 MiB/s wr, 238 op/s; 101/1593 objects degraded (6.340%); 195 KiB/s, 4 objects/s recovering 2024-02-10T06:56:13.839 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:13 smithi008 ceph-mon[66415]: osdmap e125: 8 total, 8 up, 8 in 2024-02-10T06:56:13.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:13 smithi029 ceph-mon[53210]: failed to encode map e124 with expected crc 2024-02-10T06:56:13.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:13 smithi029 ceph-mon[53210]: pgmap v231: 97 pgs: 10 peering, 5 active+undersized, 1 active+undersized+remapped, 13 active+undersized+degraded, 68 active+clean; 348 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.9 MiB/s wr, 238 op/s; 101/1593 objects degraded (6.340%); 195 KiB/s, 4 objects/s recovering 2024-02-10T06:56:13.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:13 smithi029 ceph-mon[53210]: osdmap e125: 8 total, 8 up, 8 in 2024-02-10T06:56:14.646 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:14 smithi008 ceph-mon[66415]: failed to encode map e125 with expected crc 2024-02-10T06:56:14.646 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:14 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 101/1593 objects degraded (6.340%), 13 pgs degraded (PG_DEGRADED) 2024-02-10T06:56:14.673 INFO:teuthology.orchestra.run.smithi008.stdout: 5 286794 9.69 MB/sec execute 575 sec latency 88.015 ms 2024-02-10T06:56:14.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:14 smithi029 ceph-mon[53210]: failed to encode map e125 with expected crc 2024-02-10T06:56:14.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:14 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 101/1593 objects degraded (6.340%), 13 pgs degraded (PG_DEGRADED) 2024-02-10T06:56:15.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:15 smithi008 ceph-mon[66415]: pgmap v233: 97 pgs: 4 peering, 1 active+undersized+remapped, 92 active+clean; 345 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.4 MiB/s wr, 386 op/s; 1.4 MiB/s, 34 objects/s recovering 2024-02-10T06:56:15.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:15 smithi008 ceph-mon[66415]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 101/1593 objects degraded (6.340%), 13 pgs degraded) 2024-02-10T06:56:15.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:15 smithi008 ceph-mon[66415]: Cluster is now healthy 2024-02-10T06:56:15.672 INFO:teuthology.orchestra.run.smithi008.stdout: 5 287164 9.69 MB/sec execute 576 sec latency 112.184 ms 2024-02-10T06:56:15.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:15 smithi029 ceph-mon[53210]: pgmap v233: 97 pgs: 4 peering, 1 active+undersized+remapped, 92 active+clean; 345 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.4 MiB/s wr, 386 op/s; 1.4 MiB/s, 34 objects/s recovering 2024-02-10T06:56:15.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:15 smithi029 ceph-mon[53210]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 101/1593 objects degraded (6.340%), 13 pgs degraded) 2024-02-10T06:56:15.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:15 smithi029 ceph-mon[53210]: Cluster is now healthy 2024-02-10T06:56:16.673 INFO:teuthology.orchestra.run.smithi008.stdout: 5 287528 9.68 MB/sec execute 577 sec latency 37.192 ms 2024-02-10T06:56:17.673 INFO:teuthology.orchestra.run.smithi008.stdout: 5 287855 9.69 MB/sec execute 578 sec latency 39.405 ms 2024-02-10T06:56:17.690 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:17 smithi008 ceph-mon[66415]: pgmap v234: 97 pgs: 1 peering, 1 active+undersized+remapped, 95 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.5 MiB/s wr, 295 op/s; 1.2 MiB/s, 31 objects/s recovering 2024-02-10T06:56:17.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:17 smithi029 ceph-mon[53210]: pgmap v234: 97 pgs: 1 peering, 1 active+undersized+remapped, 95 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.5 MiB/s wr, 295 op/s; 1.2 MiB/s, 31 objects/s recovering 2024-02-10T06:56:18.673 INFO:teuthology.orchestra.run.smithi008.stdout: 5 288305 9.70 MB/sec execute 579 sec latency 50.224 ms 2024-02-10T06:56:19.468 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:19 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:19.468 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:19 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:56:19.468 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:19 smithi008 ceph-mon[66415]: pgmap v235: 97 pgs: 97 active+clean; 347 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 3.0 MiB/s wr, 339 op/s; 1015 KiB/s, 25 objects/s recovering 2024-02-10T06:56:19.673 INFO:teuthology.orchestra.run.smithi008.stdout: 5 288643 9.70 MB/sec execute 580 sec latency 47.069 ms 2024-02-10T06:56:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:19 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:19 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:56:19.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:19 smithi029 ceph-mon[53210]: pgmap v235: 97 pgs: 97 active+clean; 347 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 3.0 MiB/s wr, 339 op/s; 1015 KiB/s, 25 objects/s recovering 2024-02-10T06:56:20.675 INFO:teuthology.orchestra.run.smithi008.stdout: 5 288823 9.70 MB/sec execute 581 sec latency 59.189 ms 2024-02-10T06:56:21.554 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:21 smithi008 ceph-mon[66415]: pgmap v236: 97 pgs: 97 active+clean; 347 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.0 MiB/s wr, 396 op/s; 804 KiB/s, 20 objects/s recovering 2024-02-10T06:56:21.675 INFO:teuthology.orchestra.run.smithi008.stdout: 5 289055 9.69 MB/sec execute 582 sec latency 64.201 ms 2024-02-10T06:56:21.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:21 smithi029 ceph-mon[53210]: pgmap v236: 97 pgs: 97 active+clean; 347 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.0 MiB/s wr, 396 op/s; 804 KiB/s, 20 objects/s recovering 2024-02-10T06:56:22.676 INFO:teuthology.orchestra.run.smithi008.stdout: 5 289361 9.68 MB/sec execute 583 sec latency 60.099 ms 2024-02-10T06:56:23.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:23 smithi008 ceph-mon[66415]: pgmap v237: 97 pgs: 97 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 430 op/s; 683 KiB/s, 17 objects/s recovering 2024-02-10T06:56:23.675 INFO:teuthology.orchestra.run.smithi008.stdout: 5 289698 9.68 MB/sec execute 584 sec latency 42.895 ms 2024-02-10T06:56:23.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:23 smithi029 ceph-mon[53210]: pgmap v237: 97 pgs: 97 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 430 op/s; 683 KiB/s, 17 objects/s recovering 2024-02-10T06:56:24.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:24 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:24.675 INFO:teuthology.orchestra.run.smithi008.stdout: 5 289973 9.67 MB/sec execute 585 sec latency 56.265 ms 2024-02-10T06:56:24.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:24 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:24.909 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:56:25.676 INFO:teuthology.orchestra.run.smithi008.stdout: 5 290255 9.67 MB/sec execute 586 sec latency 59.798 ms 2024-02-10T06:56:25.927 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:25 smithi008 ceph-mon[66415]: pgmap v238: 97 pgs: 97 active+clean; 351 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 4.6 MiB/s wr, 476 op/s; 134 KiB/s, 7 objects/s recovering 2024-02-10T06:56:25.927 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:25 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:56:25.927 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:25 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:56:25.927 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:25 smithi008 ceph-mon[66415]: Upgrade: osd.6 is safe to restart 2024-02-10T06:56:25.927 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:25 smithi008 ceph-mon[66415]: from='client.34414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:56:25.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:25 smithi029 ceph-mon[53210]: pgmap v238: 97 pgs: 97 active+clean; 351 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 4.6 MiB/s wr, 476 op/s; 134 KiB/s, 7 objects/s recovering 2024-02-10T06:56:25.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:25 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:56:25.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:25 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T06:56:25.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:25 smithi029 ceph-mon[53210]: Upgrade: osd.6 is safe to restart 2024-02-10T06:56:25.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:25 smithi029 ceph-mon[53210]: from='client.34414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:56:25.977 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:56:25.978 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (8m) 87s ago 18m 15.6M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:56:25.978 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (4m) 87s ago 18m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T06:56:25.978 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (4m) 21s ago 17m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T06:56:25.978 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (8m) 87s ago 18m 43.0M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:56:25.978 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (12m) 87s ago 12m 38.5M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:56:25.978 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (12m) 21s ago 12m 15.6M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:56:25.978 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (7m) 87s ago 19m 507M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:56:25.978 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (6m) 21s ago 17m 451M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:56:25.978 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (6m) 87s ago 19m 53.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:56:25.978 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (5m) 21s ago 17m 42.1M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:56:25.978 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (11m) 87s ago 11m 271M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:56:25.979 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (11m) 21s ago 11m 84.7M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:56:25.979 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (8m) 87s ago 18m 11.4M - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:56:25.979 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (7m) 21s ago 17m 17.9M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:56:25.979 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (4m) 87s ago 16m 355M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 acb0107eae6c 2024-02-10T06:56:25.979 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (3m) 87s ago 16m 282M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 09363f1c35ae 2024-02-10T06:56:25.979 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (2m) 87s ago 16m 237M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 bd6bbb022085 2024-02-10T06:56:25.979 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (97s) 87s ago 15m 15.0M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc3683d167e9 2024-02-10T06:56:25.979 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (67s) 21s ago 15m 298M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 2b1f7cbc27ce 2024-02-10T06:56:25.979 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (26s) 21s ago 14m 14.9M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 ea4052b4cafa 2024-02-10T06:56:25.979 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (14m) 21s ago 14m 622M 2705M 17.2.0 e1d6a67b021e cb99d6f851e8 2024-02-10T06:56:25.979 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (13m) 21s ago 13m 640M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:56:25.979 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (6m) 87s ago 17m 63.3M - 2.33.4 514e6a882f6e 193819b86dea 2024-02-10T06:56:26.676 INFO:teuthology.orchestra.run.smithi008.stdout: 5 290881 9.67 MB/sec execute 587 sec latency 61.134 ms 2024-02-10T06:56:26.676 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:56:26.676 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:56:26.676 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:56:26.676 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:56:26.676 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:56:26.676 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:56:26.677 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:56:26.677 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:56:26.677 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-10T06:56:26.677 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 6 2024-02-10T06:56:26.677 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:56:26.677 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:56:26.677 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:56:26.677 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:56:26.677 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:56:26.677 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-10T06:56:26.677 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 10 2024-02-10T06:56:26.677 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:56:26.677 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:56:26.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:26 smithi008 ceph-mon[66415]: from='client.44315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:56:26.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:26 smithi008 ceph-mon[66415]: from='client.44319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:56:26.791 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:26 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/258869685' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:56:26.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:26 smithi029 ceph-mon[53210]: from='client.44315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:56:26.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:26 smithi029 ceph-mon[53210]: from='client.44319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:56:26.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:26 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/258869685' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:56:27.252 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:56:27.253 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:56:27.253 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:56:27.253 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:56:27.253 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:56:27.253 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:56:27.253 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-02-10T06:56:27.254 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-02-10T06:56:27.254 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:56:27.254 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "12/23 daemons upgraded", 2024-02-10T06:56:27.254 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:56:27.254 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:56:27.254 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:56:27.676 INFO:teuthology.orchestra.run.smithi008.stdout: 5 291417 9.69 MB/sec execute 588 sec latency 33.596 ms 2024-02-10T06:56:27.923 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:56:27.960 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:27 smithi008 ceph-mon[66415]: pgmap v239: 97 pgs: 97 active+clean; 352 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.0 MiB/s wr, 353 op/s; 108 KiB/s, 2 objects/s recovering 2024-02-10T06:56:27.960 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:27.960 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T06:56:27.960 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:56:27.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:27 smithi029 ceph-mon[53210]: pgmap v239: 97 pgs: 97 active+clean; 352 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.0 MiB/s wr, 353 op/s; 108 KiB/s, 2 objects/s recovering 2024-02-10T06:56:27.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:27.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T06:56:27.976 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:56:28.676 INFO:teuthology.orchestra.run.smithi008.stdout: 5 291932 9.70 MB/sec execute 589 sec latency 40.336 ms 2024-02-10T06:56:28.738 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:28 smithi008 ceph-mon[66415]: Upgrade: Updating osd.6 2024-02-10T06:56:28.738 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:28 smithi008 ceph-mon[66415]: Deploying daemon osd.6 on smithi029 2024-02-10T06:56:28.738 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:28 smithi008 ceph-mon[66415]: from='client.34430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:56:28.738 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:28 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/1107752335' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:56:28.980 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:28 smithi029 ceph-mon[53210]: Upgrade: Updating osd.6 2024-02-10T06:56:28.981 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:28 smithi029 ceph-mon[53210]: Deploying daemon osd.6 on smithi029 2024-02-10T06:56:28.981 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:28 smithi029 ceph-mon[53210]: from='client.34430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:56:28.981 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:28 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/1107752335' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:56:29.676 INFO:teuthology.orchestra.run.smithi008.stdout: 5 292226 9.70 MB/sec execute 590 sec latency 63.902 ms 2024-02-10T06:56:29.752 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:29 smithi008 ceph-mon[66415]: pgmap v240: 97 pgs: 97 active+clean; 357 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 5.0 MiB/s wr, 547 op/s 2024-02-10T06:56:29.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:29 smithi029 ceph-mon[53210]: pgmap v240: 97 pgs: 97 active+clean; 357 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 5.0 MiB/s wr, 547 op/s 2024-02-10T06:56:30.676 INFO:teuthology.orchestra.run.smithi008.stdout: 5 292375 9.69 MB/sec execute 591 sec latency 92.055 ms 2024-02-10T06:56:31.676 INFO:teuthology.orchestra.run.smithi008.stdout: 5 292553 9.69 MB/sec execute 592 sec latency 56.361 ms 2024-02-10T06:56:31.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:31 smithi008 ceph-mon[66415]: pgmap v241: 97 pgs: 97 active+clean; 350 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.6 MiB/s wr, 555 op/s 2024-02-10T06:56:31.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:31 smithi029 ceph-mon[53210]: pgmap v241: 97 pgs: 97 active+clean; 350 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.6 MiB/s wr, 555 op/s 2024-02-10T06:56:31.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:31 smithi029 systemd[1]: Stopping Ceph osd.6 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:56:32.676 INFO:teuthology.orchestra.run.smithi008.stdout: 5 292715 9.67 MB/sec execute 593 sec latency 88.212 ms 2024-02-10T06:56:33.208 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:32 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[37262]: 2024-02-10T06:56:32.787+0000 7f3a97436700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:56:33.208 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:32 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[37262]: 2024-02-10T06:56:32.787+0000 7f3a97436700 -1 osd.6 125 *** Got signal Terminated *** 2024-02-10T06:56:33.208 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:32 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[37262]: 2024-02-10T06:56:32.787+0000 7f3a97436700 -1 osd.6 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:56:33.676 INFO:teuthology.orchestra.run.smithi008.stdout: 5 293025 9.67 MB/sec execute 594 sec latency 62.700 ms 2024-02-10T06:56:34.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:33 smithi008 ceph-mon[66415]: pgmap v242: 97 pgs: 97 active+clean; 351 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.9 MiB/s wr, 494 op/s 2024-02-10T06:56:34.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:33 smithi008 ceph-mon[66415]: osd.6 marked itself down and dead 2024-02-10T06:56:34.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:33 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:56:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:33 smithi029 ceph-mon[53210]: pgmap v242: 97 pgs: 97 active+clean; 351 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.9 MiB/s wr, 494 op/s 2024-02-10T06:56:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:33 smithi029 ceph-mon[53210]: osd.6 marked itself down and dead 2024-02-10T06:56:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:33 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:56:34.676 INFO:teuthology.orchestra.run.smithi008.stdout: 5 293030 9.65 MB/sec execute 595 sec latency 970.394 ms 2024-02-10T06:56:34.682 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:34 smithi029 podman[62202]: 2024-02-10 06:56:34.409620984 +0000 UTC m=+2.414577917 container died cb99d6f851e84b8c5399fb2bd42d520e6396fc72ce496bcc74f50d9e18c46852 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, maintainer=Guillaume Abrioux , ceph=True, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8) 2024-02-10T06:56:34.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:34 smithi029 podman[62202]: 2024-02-10 06:56:34.782710329 +0000 UTC m=+2.787667261 container cleanup cb99d6f851e84b8c5399fb2bd42d520e6396fc72ce496bcc74f50d9e18c46852 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, distribution-scope=public, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, ceph=True, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, version=8, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.openshift.expose-services=, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc.) 2024-02-10T06:56:34.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:34 smithi029 bash[62202]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6 2024-02-10T06:56:34.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:34 smithi029 ceph-mon[53210]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:56:34.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:34 smithi029 ceph-mon[53210]: osdmap e126: 8 total, 7 up, 8 in 2024-02-10T06:56:34.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:34 smithi029 ceph-mon[53210]: failed to encode map e126 with expected crc 2024-02-10T06:56:34.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:34 smithi029 ceph-mon[53210]: failed to encode map e126 with expected crc 2024-02-10T06:56:34.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:34 smithi029 ceph-mon[53210]: failed to encode map e126 with expected crc 2024-02-10T06:56:34.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:34 smithi029 ceph-mon[53210]: failed to encode map e126 with expected crc 2024-02-10T06:56:34.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:34 smithi008 ceph-mon[66415]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:56:34.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:34 smithi008 ceph-mon[66415]: osdmap e126: 8 total, 7 up, 8 in 2024-02-10T06:56:34.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:34 smithi008 ceph-mon[66415]: failed to encode map e126 with expected crc 2024-02-10T06:56:34.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:34 smithi008 ceph-mon[66415]: failed to encode map e126 with expected crc 2024-02-10T06:56:34.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:34 smithi008 ceph-mon[66415]: failed to encode map e126 with expected crc 2024-02-10T06:56:34.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:34 smithi008 ceph-mon[66415]: failed to encode map e126 with expected crc 2024-02-10T06:56:35.458 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:35 smithi029 podman[62217]: 2024-02-10 06:56:35.137986141 +0000 UTC m=+0.729878960 container remove cb99d6f851e84b8c5399fb2bd42d520e6396fc72ce496bcc74f50d9e18c46852 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6, io.openshift.tags=base centos centos-stream, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, version=8, name=centos-stream, ceph=True, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-10T06:56:35.676 INFO:teuthology.orchestra.run.smithi008.stdout: 5 293373 9.65 MB/sec execute 596 sec latency 1066.259 ms 2024-02-10T06:56:35.746 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:35 smithi029 podman[62309]: 2024-02-10 06:56:35.397896239 +0000 UTC m=+0.021080738 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:56:35.747 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:35 smithi029 podman[62309]: 2024-02-10 06:56:35.523598024 +0000 UTC m=+0.146782558 container create 144b6cab1d9efd669ffcd6a7d387330356b0ae7955918a567be1c9fa6ea36a28 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:56:35.762 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:35 smithi008 ceph-mon[66415]: pgmap v244: 97 pgs: 8 peering, 89 active+clean; 354 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.2 MiB/s wr, 540 op/s 2024-02-10T06:56:35.762 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:35 smithi008 ceph-mon[66415]: failed to encode map e127 with expected crc 2024-02-10T06:56:35.762 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:35 smithi008 ceph-mon[66415]: osdmap e127: 8 total, 7 up, 8 in 2024-02-10T06:56:35.762 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:35 smithi008 ceph-mon[66415]: failed to encode map e127 with expected crc 2024-02-10T06:56:36.054 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:35 smithi029 ceph-mon[53210]: pgmap v244: 97 pgs: 8 peering, 89 active+clean; 354 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.2 MiB/s wr, 540 op/s 2024-02-10T06:56:36.054 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:35 smithi029 ceph-mon[53210]: failed to encode map e127 with expected crc 2024-02-10T06:56:36.054 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:35 smithi029 ceph-mon[53210]: osdmap e127: 8 total, 7 up, 8 in 2024-02-10T06:56:36.054 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:35 smithi029 ceph-mon[53210]: failed to encode map e127 with expected crc 2024-02-10T06:56:36.340 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:36 smithi029 podman[62309]: 2024-02-10 06:56:36.05378379 +0000 UTC m=+0.676968292 container init 144b6cab1d9efd669ffcd6a7d387330356b0ae7955918a567be1c9fa6ea36a28 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_CLEAN=True, io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-10T06:56:36.340 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:36 smithi029 podman[62309]: 2024-02-10 06:56:36.057766044 +0000 UTC m=+0.680950548 container start 144b6cab1d9efd669ffcd6a7d387330356b0ae7955918a567be1c9fa6ea36a28 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.schema-version=1.0, ceph=True) 2024-02-10T06:56:36.676 INFO:teuthology.orchestra.run.smithi008.stdout: 5 293713 9.65 MB/sec execute 597 sec latency 43.411 ms 2024-02-10T06:56:36.708 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:36 smithi029 podman[62309]: 2024-02-10 06:56:36.340128074 +0000 UTC m=+0.963312590 container attach 144b6cab1d9efd669ffcd6a7d387330356b0ae7955918a567be1c9fa6ea36a28 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-290212a) 2024-02-10T06:56:36.708 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:36 smithi029 podman[62309]: 2024-02-10 06:56:36.402534136 +0000 UTC m=+1.025718635 container died 144b6cab1d9efd669ffcd6a7d387330356b0ae7955918a567be1c9fa6ea36a28 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:56:37.026 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:36 smithi008 ceph-mon[66415]: Health check failed: Degraded data redundancy: 11/1656 objects degraded (0.664%), 1 pg degraded (PG_DEGRADED) 2024-02-10T06:56:37.101 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:36 smithi029 ceph-mon[53210]: Health check failed: Degraded data redundancy: 11/1656 objects degraded (0.664%), 1 pg degraded (PG_DEGRADED) 2024-02-10T06:56:37.359 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:37 smithi029 podman[62334]: 2024-02-10 06:56:37.101066074 +0000 UTC m=+0.690181733 container remove 144b6cab1d9efd669ffcd6a7d387330356b0ae7955918a567be1c9fa6ea36a28 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-290212a) 2024-02-10T06:56:37.359 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:37 smithi029 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.6.service: Deactivated successfully. 2024-02-10T06:56:37.359 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:37 smithi029 systemd[1]: Stopped Ceph osd.6 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:56:37.359 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:37 smithi029 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.6.service: Consumed 5min 55.462s CPU time. 2024-02-10T06:56:37.609 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:37 smithi029 systemd[1]: Starting Ceph osd.6 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:56:37.676 INFO:teuthology.orchestra.run.smithi008.stdout: 5 294079 9.65 MB/sec execute 598 sec latency 83.889 ms 2024-02-10T06:56:37.894 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:37 smithi029 ceph-mon[53210]: pgmap v246: 97 pgs: 1 active+undersized+degraded, 8 stale+active+clean, 8 peering, 80 active+clean; 354 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.4 MiB/s wr, 358 op/s; 11/1656 objects degraded (0.664%) 2024-02-10T06:56:37.894 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:37 smithi029 podman[62433]: 2024-02-10 06:56:37.513508823 +0000 UTC m=+0.022730973 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:56:37.894 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:37 smithi029 podman[62433]: 2024-02-10 06:56:37.620004039 +0000 UTC m=+0.129226174 container create 99e44b87c802c1d74e1f56e22c99e895dce0f4b092c0328d1330e2dd2c51fa41 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-activate, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-02-10T06:56:37.894 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:37 smithi029 podman[62433]: 2024-02-10 06:56:37.89385876 +0000 UTC m=+0.403080903 container init 99e44b87c802c1d74e1f56e22c99e895dce0f4b092c0328d1330e2dd2c51fa41 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-activate, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-10T06:56:38.049 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:37 smithi008 ceph-mon[66415]: pgmap v246: 97 pgs: 1 active+undersized+degraded, 8 stale+active+clean, 8 peering, 80 active+clean; 354 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.4 MiB/s wr, 358 op/s; 11/1656 objects degraded (0.664%) 2024-02-10T06:56:38.208 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:37 smithi029 podman[62433]: 2024-02-10 06:56:37.898756264 +0000 UTC m=+0.407978404 container start 99e44b87c802c1d74e1f56e22c99e895dce0f4b092c0328d1330e2dd2c51fa41 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-activate, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-10T06:56:38.208 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:38 smithi029 podman[62433]: 2024-02-10 06:56:38.011238134 +0000 UTC m=+0.520460276 container attach 99e44b87c802c1d74e1f56e22c99e895dce0f4b092c0328d1330e2dd2c51fa41 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-10T06:56:38.676 INFO:teuthology.orchestra.run.smithi008.stdout: 5 294464 9.64 MB/sec execute 599 sec latency 144.917 ms 2024-02-10T06:56:38.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:38 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-activate[62448]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T06:56:38.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:38 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-activate[62448]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-10T06:56:38.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:38 smithi029 bash[62433]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T06:56:38.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:38 smithi029 bash[62433]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-10T06:56:38.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:38 smithi029 bash[62433]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-10T06:56:38.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:38 smithi029 bash[62433]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T06:56:38.959 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:38 smithi029 bash[62433]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-10T06:56:38.959 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:38 smithi029 bash[62433]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T06:56:38.959 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:38 smithi029 bash[62433]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-10T06:56:38.959 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:38 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-activate[62448]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-10T06:56:38.959 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:38 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-activate[62448]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T06:56:38.959 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:38 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-activate[62448]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-10T06:56:38.959 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:38 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-activate[62448]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T06:56:38.959 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:38 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-activate[62448]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-10T06:56:38.959 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:38 smithi029 podman[62433]: 2024-02-10 06:56:38.507829418 +0000 UTC m=+1.017051558 container died 99e44b87c802c1d74e1f56e22c99e895dce0f4b092c0328d1330e2dd2c51fa41 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-activate, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-10T06:56:39.676 INFO:teuthology.orchestra.run.smithi008.stdout: 5 cleanup 600 sec 2024-02-10T06:56:39.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:39 smithi008 ceph-mon[66415]: pgmap v247: 97 pgs: 9 active+undersized, 18 active+undersized+degraded, 8 peering, 62 active+clean; 350 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.7 MiB/s wr, 419 op/s; 124/1590 objects degraded (7.799%) 2024-02-10T06:56:39.891 INFO:teuthology.orchestra.run.smithi008.stdout: 0 cleanup 600 sec 2024-02-10T06:56:39.891 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:56:39.891 INFO:teuthology.orchestra.run.smithi008.stdout: Operation Count AvgLat MaxLat 2024-02-10T06:56:39.891 INFO:teuthology.orchestra.run.smithi008.stdout: ---------------------------------------- 2024-02-10T06:56:39.891 INFO:teuthology.orchestra.run.smithi008.stdout: NTCreateX 185339 4.664 1038.301 2024-02-10T06:56:39.891 INFO:teuthology.orchestra.run.smithi008.stdout: Close 136124 0.141 24.766 2024-02-10T06:56:39.891 INFO:teuthology.orchestra.run.smithi008.stdout: Rename 7853 10.184 176.055 2024-02-10T06:56:39.891 INFO:teuthology.orchestra.run.smithi008.stdout: Unlink 37467 7.144 291.565 2024-02-10T06:56:39.892 INFO:teuthology.orchestra.run.smithi008.stdout: Qpathinfo 168189 1.184 165.682 2024-02-10T06:56:39.892 INFO:teuthology.orchestra.run.smithi008.stdout: Qfileinfo 29322 0.008 2.128 2024-02-10T06:56:39.892 INFO:teuthology.orchestra.run.smithi008.stdout: Qfsinfo 30834 0.624 24.506 2024-02-10T06:56:39.892 INFO:teuthology.orchestra.run.smithi008.stdout: Sfileinfo 15097 12.933 204.154 2024-02-10T06:56:39.892 INFO:teuthology.orchestra.run.smithi008.stdout: Find 65021 1.019 27.083 2024-02-10T06:56:39.892 INFO:teuthology.orchestra.run.smithi008.stdout: WriteX 91624 13.991 2201.622 2024-02-10T06:56:39.892 INFO:teuthology.orchestra.run.smithi008.stdout: ReadX 291004 0.010 4.152 2024-02-10T06:56:39.892 INFO:teuthology.orchestra.run.smithi008.stdout: LockX 604 0.396 2.669 2024-02-10T06:56:39.892 INFO:teuthology.orchestra.run.smithi008.stdout: UnlockX 604 0.345 4.010 2024-02-10T06:56:39.892 INFO:teuthology.orchestra.run.smithi008.stdout: Flush 12997 0.002 0.035 2024-02-10T06:56:39.892 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:56:39.892 INFO:teuthology.orchestra.run.smithi008.stdout:Throughput 9.63982 MB/sec 5 clients 5 procs max_latency=2201.630 ms 2024-02-10T06:56:39.893 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'umount /mnt/foo' 2024-02-10T06:56:39.933 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done' 2024-02-10T06:56:40.062 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5' 2024-02-10T06:56:40.075 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:39 smithi029 ceph-mon[53210]: pgmap v247: 97 pgs: 9 active+undersized, 18 active+undersized+degraded, 8 peering, 62 active+clean; 350 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.7 MiB/s wr, 419 op/s; 124/1590 objects degraded (7.799%) 2024-02-10T06:56:40.089 INFO:teuthology.orchestra.run.smithi008.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-02-10T06:56:40.089 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:56:40.089 INFO:teuthology.orchestra.run.smithi008.stdout:Running for 5 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 1 secs 2024-02-10T06:56:40.089 INFO:teuthology.orchestra.run.smithi008.stdout:3 of 5 processes prepared for launch 0 sec 2024-02-10T06:56:40.189 INFO:teuthology.orchestra.run.smithi008.stdout:5 of 5 processes prepared for launch 0 sec 2024-02-10T06:56:40.190 INFO:teuthology.orchestra.run.smithi008.stdout:releasing clients 2024-02-10T06:56:40.619 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:40 smithi029 podman[62574]: 2024-02-10 06:56:40.389071591 +0000 UTC m=+1.874379377 container remove 99e44b87c802c1d74e1f56e22c99e895dce0f4b092c0328d1330e2dd2c51fa41 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-activate, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-10T06:56:40.881 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:40 smithi029 podman[62635]: 2024-02-10 06:56:40.521918014 +0000 UTC m=+0.021677093 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:56:40.881 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:40 smithi029 podman[62635]: 2024-02-10 06:56:40.624455 +0000 UTC m=+0.124214067 container create d89cf7ba8295c30314e43a79b85e649b63eb0b5a66afe4a91b45bc69b350b6b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-02-10T06:56:41.168 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:40 smithi029 podman[62635]: 2024-02-10 06:56:40.881506518 +0000 UTC m=+0.381265586 container init d89cf7ba8295c30314e43a79b85e649b63eb0b5a66afe4a91b45bc69b350b6b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=) 2024-02-10T06:56:41.169 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:40 smithi029 podman[62635]: 2024-02-10 06:56:40.885031588 +0000 UTC m=+0.384790659 container start d89cf7ba8295c30314e43a79b85e649b63eb0b5a66afe4a91b45bc69b350b6b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, RELEASE=main-290212a, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-02-10T06:56:41.169 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:40 smithi029 ceph-osd[62653]: -- 172.21.15.29:0/480925365 <== mon.1 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b2ef8831e0 con 0x55b2ef855000 2024-02-10T06:56:41.169 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:40 smithi029 bash[62635]: d89cf7ba8295c30314e43a79b85e649b63eb0b5a66afe4a91b45bc69b350b6b3 2024-02-10T06:56:41.169 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:41 smithi029 systemd[1]: Started Ceph osd.6 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:56:41.707 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:41 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T06:56:41.430+0000 7f5cd25947c0 -1 Falling back to public interface 2024-02-10T06:56:42.012 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:41 smithi008 ceph-mon[66415]: pgmap v248: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 343 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 462 op/s; 170/1536 objects degraded (11.068%) 2024-02-10T06:56:42.013 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:41 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:42.013 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:41 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:42.013 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:41 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:56:42.190 INFO:teuthology.orchestra.run.smithi008.stdout: 5 391 22.11 MB/sec execute 1 sec latency 52.403 ms 2024-02-10T06:56:42.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:41 smithi029 ceph-mon[53210]: pgmap v248: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 343 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 462 op/s; 170/1536 objects degraded (11.068%) 2024-02-10T06:56:42.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:41 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:42.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:41 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:42.198 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:41 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:56:43.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:42 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 181/1608 objects degraded (11.256%), 25 pgs degraded (PG_DEGRADED) 2024-02-10T06:56:43.127 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:42 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 181/1608 objects degraded (11.256%), 25 pgs degraded (PG_DEGRADED) 2024-02-10T06:56:43.190 INFO:teuthology.orchestra.run.smithi008.stdout: 5 593 26.08 MB/sec execute 2 sec latency 103.909 ms 2024-02-10T06:56:44.048 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:43 smithi008 ceph-mon[66415]: pgmap v249: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 356 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 5.0 MiB/s wr, 385 op/s; 181/1608 objects degraded (11.256%) 2024-02-10T06:56:44.190 INFO:teuthology.orchestra.run.smithi008.stdout: 5 887 20.36 MB/sec execute 3 sec latency 37.932 ms 2024-02-10T06:56:44.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:43 smithi029 ceph-mon[53210]: pgmap v249: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 356 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 5.0 MiB/s wr, 385 op/s; 181/1608 objects degraded (11.256%) 2024-02-10T06:56:45.190 INFO:teuthology.orchestra.run.smithi008.stdout: 5 1228 17.02 MB/sec execute 4 sec latency 40.140 ms 2024-02-10T06:56:46.080 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:45 smithi008 ceph-mon[66415]: pgmap v250: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 8.7 MiB/s wr, 417 op/s; 155/1377 objects degraded (11.256%) 2024-02-10T06:56:46.080 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:45 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:46.080 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:45 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:46.190 INFO:teuthology.orchestra.run.smithi008.stdout: 5 cleanup 5 sec 2024-02-10T06:56:46.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:45 smithi029 ceph-mon[53210]: pgmap v250: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 8.7 MiB/s wr, 417 op/s; 155/1377 objects degraded (11.256%) 2024-02-10T06:56:46.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:45 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:46.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:45 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:46.347 INFO:teuthology.orchestra.run.smithi008.stdout:[1705] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000cd5200000027' failed - Device or resource busy 2024-02-10T06:56:46.363 INFO:teuthology.orchestra.run.smithi008.stdout:[1705] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000cd5c0000000e' failed - Device or resource busy 2024-02-10T06:56:46.440 INFO:teuthology.orchestra.run.smithi008.stdout: 0 cleanup 5 sec 2024-02-10T06:56:46.441 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:56:46.441 INFO:teuthology.orchestra.run.smithi008.stdout: Operation Count AvgLat MaxLat 2024-02-10T06:56:46.441 INFO:teuthology.orchestra.run.smithi008.stdout: ---------------------------------------- 2024-02-10T06:56:46.441 INFO:teuthology.orchestra.run.smithi008.stdout: NTCreateX 1110 5.913 52.396 2024-02-10T06:56:46.441 INFO:teuthology.orchestra.run.smithi008.stdout: Close 925 6.246 103.900 2024-02-10T06:56:46.441 INFO:teuthology.orchestra.run.smithi008.stdout: Rename 51 11.576 40.133 2024-02-10T06:56:46.441 INFO:teuthology.orchestra.run.smithi008.stdout: Unlink 106 5.647 42.954 2024-02-10T06:56:46.441 INFO:teuthology.orchestra.run.smithi008.stdout: Qpathinfo 739 1.744 31.983 2024-02-10T06:56:46.441 INFO:teuthology.orchestra.run.smithi008.stdout: Qfileinfo 388 14.684 55.791 2024-02-10T06:56:46.441 INFO:teuthology.orchestra.run.smithi008.stdout: Qfsinfo 165 0.423 1.480 2024-02-10T06:56:46.441 INFO:teuthology.orchestra.run.smithi008.stdout: Sfileinfo 170 13.632 29.263 2024-02-10T06:56:46.441 INFO:teuthology.orchestra.run.smithi008.stdout: Find 301 0.791 2.391 2024-02-10T06:56:46.441 INFO:teuthology.orchestra.run.smithi008.stdout: WriteX 1663 0.023 0.073 2024-02-10T06:56:46.441 INFO:teuthology.orchestra.run.smithi008.stdout: ReadX 1272 0.013 0.512 2024-02-10T06:56:46.441 INFO:teuthology.orchestra.run.smithi008.stdout: Flush 133 13.968 31.196 2024-02-10T06:56:46.442 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T06:56:46.442 INFO:teuthology.orchestra.run.smithi008.stdout:Throughput 17.024 MB/sec 5 clients 5 procs max_latency=103.909 ms 2024-02-10T06:56:46.443 DEBUG:teuthology.parallel:result is None 2024-02-10T06:56:47.805 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:47 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T06:56:47.592+0000 7f5cd25947c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-02-10T06:56:47.805 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:47 smithi029 ceph-mon[53210]: pgmap v251: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 9.7 MiB/s wr, 381 op/s; 158/1371 objects degraded (11.524%) 2024-02-10T06:56:47.805 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:47.805 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:48.029 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:47 smithi008 ceph-mon[66415]: pgmap v251: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 9.7 MiB/s wr, 381 op/s; 158/1371 objects degraded (11.524%) 2024-02-10T06:56:48.029 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:48.029 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:48.975 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:48 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T06:56:48.885+0000 7f5cd25947c0 -1 osd.6 125 log_to_monitors true 2024-02-10T06:56:49.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:49.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:56:49.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:49 smithi029 ceph-mon[53210]: pgmap v252: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 343 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 12 MiB/s wr, 499 op/s; 169/1413 objects degraded (11.960%) 2024-02-10T06:56:49.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:49 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 158/1371 objects degraded (11.524%), 25 pgs degraded (PG_DEGRADED) 2024-02-10T06:56:49.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:49 smithi029 ceph-mon[53210]: from='osd.6 [v2:172.21.15.29:6816/1324962279,v1:172.21.15.29:6817/1324962279]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T06:56:49.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:49.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:56:49.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:49 smithi008 ceph-mon[66415]: pgmap v252: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 343 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 12 MiB/s wr, 499 op/s; 169/1413 objects degraded (11.960%) 2024-02-10T06:56:49.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:49 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 158/1371 objects degraded (11.524%), 25 pgs degraded (PG_DEGRADED) 2024-02-10T06:56:49.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:49 smithi008 ceph-mon[66415]: from='osd.6 [v2:172.21.15.29:6816/1324962279,v1:172.21.15.29:6817/1324962279]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T06:56:50.208 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 06:56:49 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T06:56:49.723+0000 7f5cc39ec700 -1 osd.6 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:56:51.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:50 smithi008 ceph-mon[66415]: from='osd.6 [v2:172.21.15.29:6816/1324962279,v1:172.21.15.29:6817/1324962279]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T06:56:51.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:50 smithi008 ceph-mon[66415]: osdmap e128: 8 total, 7 up, 8 in 2024-02-10T06:56:51.035 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:50 smithi008 ceph-mon[66415]: from='osd.6 [v2:172.21.15.29:6816/1324962279,v1:172.21.15.29:6817/1324962279]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:56:51.077 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:50 smithi029 ceph-mon[53210]: from='osd.6 [v2:172.21.15.29:6816/1324962279,v1:172.21.15.29:6817/1324962279]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T06:56:51.077 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:50 smithi029 ceph-mon[53210]: osdmap e128: 8 total, 7 up, 8 in 2024-02-10T06:56:51.077 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:50 smithi029 ceph-mon[53210]: from='osd.6 [v2:172.21.15.29:6816/1324962279,v1:172.21.15.29:6817/1324962279]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:56:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: failed to encode map e128 with expected crc 2024-02-10T06:56:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: pgmap v254: 97 pgs: 11 active+undersized, 24 active+undersized+degraded, 62 active+clean; 301 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 13 MiB/s wr, 446 op/s; 133/984 objects degraded (13.516%) 2024-02-10T06:56:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:56:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: osd.6 [v2:172.21.15.29:6816/1324962279,v1:172.21.15.29:6817/1324962279] boot 2024-02-10T06:56:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: osdmap e129: 8 total, 8 up, 8 in 2024-02-10T06:56:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:56:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: failed to encode map e129 with expected crc 2024-02-10T06:56:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:52.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:52.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:56:52.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:56:52.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:52.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:52.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:56:52.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:56:52.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:56:52.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:56:52.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:56:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: failed to encode map e128 with expected crc 2024-02-10T06:56:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: pgmap v254: 97 pgs: 11 active+undersized, 24 active+undersized+degraded, 62 active+clean; 301 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 13 MiB/s wr, 446 op/s; 133/984 objects degraded (13.516%) 2024-02-10T06:56:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:56:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: osd.6 [v2:172.21.15.29:6816/1324962279,v1:172.21.15.29:6817/1324962279] boot 2024-02-10T06:56:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: osdmap e129: 8 total, 8 up, 8 in 2024-02-10T06:56:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T06:56:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: failed to encode map e129 with expected crc 2024-02-10T06:56:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:52.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:52.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:56:52.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:56:52.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:52.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:56:52.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:56:52.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:56:52.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:56:52.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:56:52.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:56:53.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:52 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:56:53.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:52 smithi008 ceph-mon[66415]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-10T06:56:53.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:52 smithi008 ceph-mon[66415]: osdmap e130: 8 total, 8 up, 8 in 2024-02-10T06:56:53.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:52 smithi008 ceph-mon[66415]: failed to encode map e130 with expected crc 2024-02-10T06:56:53.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:52 smithi008 ceph-mon[66415]: failed to encode map e130 with expected crc 2024-02-10T06:56:53.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:52 smithi008 ceph-mon[66415]: failed to encode map e130 with expected crc 2024-02-10T06:56:53.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:52 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:56:53.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:52 smithi029 ceph-mon[53210]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-10T06:56:53.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:52 smithi029 ceph-mon[53210]: osdmap e130: 8 total, 8 up, 8 in 2024-02-10T06:56:53.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:52 smithi029 ceph-mon[53210]: failed to encode map e130 with expected crc 2024-02-10T06:56:53.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:52 smithi029 ceph-mon[53210]: failed to encode map e130 with expected crc 2024-02-10T06:56:53.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:52 smithi029 ceph-mon[53210]: failed to encode map e130 with expected crc 2024-02-10T06:56:54.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:53 smithi008 ceph-mon[66415]: pgmap v257: 97 pgs: 1 active+recovery_wait+degraded, 3 remapped+peering, 9 active+undersized, 19 active+undersized+degraded, 65 active+clean; 276 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 5.4 MiB/s wr, 338 op/s; 111/774 objects degraded (14.341%) 2024-02-10T06:56:54.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:53 smithi008 ceph-mon[66415]: osdmap e131: 8 total, 8 up, 8 in 2024-02-10T06:56:54.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:53 smithi008 ceph-mon[66415]: failed to encode map e131 with expected crc 2024-02-10T06:56:54.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:53 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 111/774 objects degraded (14.341%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T06:56:54.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:53 smithi008 ceph-mon[66415]: osdmap e132: 8 total, 8 up, 8 in 2024-02-10T06:56:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:53 smithi029 ceph-mon[53210]: pgmap v257: 97 pgs: 1 active+recovery_wait+degraded, 3 remapped+peering, 9 active+undersized, 19 active+undersized+degraded, 65 active+clean; 276 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 5.4 MiB/s wr, 338 op/s; 111/774 objects degraded (14.341%) 2024-02-10T06:56:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:53 smithi029 ceph-mon[53210]: osdmap e131: 8 total, 8 up, 8 in 2024-02-10T06:56:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:53 smithi029 ceph-mon[53210]: failed to encode map e131 with expected crc 2024-02-10T06:56:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:53 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 111/774 objects degraded (14.341%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T06:56:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:53 smithi029 ceph-mon[53210]: osdmap e132: 8 total, 8 up, 8 in 2024-02-10T06:56:55.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:54 smithi008 ceph-mon[66415]: failed to encode map e132 with expected crc 2024-02-10T06:56:55.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:54 smithi008 ceph-mon[66415]: failed to encode map e132 with expected crc 2024-02-10T06:56:55.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:54 smithi008 ceph-mon[66415]: osdmap e133: 8 total, 8 up, 8 in 2024-02-10T06:56:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:54 smithi029 ceph-mon[53210]: failed to encode map e132 with expected crc 2024-02-10T06:56:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:54 smithi029 ceph-mon[53210]: failed to encode map e132 with expected crc 2024-02-10T06:56:55.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:54 smithi029 ceph-mon[53210]: osdmap e133: 8 total, 8 up, 8 in 2024-02-10T06:56:56.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:55 smithi008 ceph-mon[66415]: pgmap v260: 97 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 remapped+peering, 1 active+undersized, 6 active+undersized+degraded, 82 active+clean; 242 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 174 KiB/s wr, 46 op/s; 108/474 objects degraded (22.785%); 0 B/s, 4 keys/s, 34 objects/s recovering 2024-02-10T06:56:56.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:55 smithi008 ceph-mon[66415]: failed to encode map e133 with expected crc 2024-02-10T06:56:56.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:55 smithi008 ceph-mon[66415]: failed to encode map e133 with expected crc 2024-02-10T06:56:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:55 smithi029 ceph-mon[53210]: pgmap v260: 97 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 remapped+peering, 1 active+undersized, 6 active+undersized+degraded, 82 active+clean; 242 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 174 KiB/s wr, 46 op/s; 108/474 objects degraded (22.785%); 0 B/s, 4 keys/s, 34 objects/s recovering 2024-02-10T06:56:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:55 smithi029 ceph-mon[53210]: failed to encode map e133 with expected crc 2024-02-10T06:56:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:55 smithi029 ceph-mon[53210]: failed to encode map e133 with expected crc 2024-02-10T06:56:58.029 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:57 smithi008 ceph-mon[66415]: pgmap v262: 97 pgs: 3 active+recovery_wait+degraded, 2 remapped+peering, 92 active+clean; 238 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 150 KiB/s wr, 43 op/s; 86/438 objects degraded (19.635%); 2.3 MiB/s, 4 keys/s, 43 objects/s recovering 2024-02-10T06:56:58.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:57 smithi029 ceph-mon[53210]: pgmap v262: 97 pgs: 3 active+recovery_wait+degraded, 2 remapped+peering, 92 active+clean; 238 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 150 KiB/s wr, 43 op/s; 86/438 objects degraded (19.635%); 2.3 MiB/s, 4 keys/s, 43 objects/s recovering 2024-02-10T06:56:58.284 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:56:58.956 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:56:58.956 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (9m) 2m ago 19m 15.6M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:56:58.956 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (5m) 2m ago 18m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T06:56:58.957 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (5m) 13s ago 18m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T06:56:58.957 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (8m) 2m ago 18m 43.0M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:56:58.957 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (12m) 2m ago 12m 38.5M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:56:58.957 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (12m) 13s ago 12m 15.7M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:56:58.957 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (8m) 2m ago 20m 507M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:56:58.957 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (7m) 13s ago 18m 452M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:56:58.957 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (6m) 2m ago 20m 53.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:56:58.957 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (6m) 13s ago 18m 44.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:56:58.957 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (12m) 2m ago 12m 271M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:56:58.957 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (12m) 13s ago 12m 84.7M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:56:58.957 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (8m) 2m ago 18m 11.4M - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:56:58.957 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (8m) 13s ago 18m 20.2M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:56:58.957 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (4m) 2m ago 17m 355M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 acb0107eae6c 2024-02-10T06:56:58.957 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (3m) 2m ago 17m 282M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 09363f1c35ae 2024-02-10T06:56:58.958 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (2m) 2m ago 16m 237M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 bd6bbb022085 2024-02-10T06:56:58.958 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (2m) 2m ago 16m 15.0M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc3683d167e9 2024-02-10T06:56:58.958 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (100s) 13s ago 15m 340M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 2b1f7cbc27ce 2024-02-10T06:56:58.958 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (59s) 13s ago 15m 300M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 ea4052b4cafa 2024-02-10T06:56:58.958 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (18s) 13s ago 14m 14.3M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 d89cf7ba8295 2024-02-10T06:56:58.958 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (14m) 13s ago 14m 697M 2705M 17.2.0 e1d6a67b021e badc7a3ebb0c 2024-02-10T06:56:58.958 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (7m) 2m ago 17m 63.3M - 2.33.4 514e6a882f6e 193819b86dea 2024-02-10T06:56:58.988 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:58 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 86/438 objects degraded (19.635%), 3 pgs degraded (PG_DEGRADED) 2024-02-10T06:56:59.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:58 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 86/438 objects degraded (19.635%), 3 pgs degraded (PG_DEGRADED) 2024-02-10T06:56:59.363 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:56:59.364 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:56:59.364 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:56:59.364 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:56:59.364 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:56:59.364 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:56:59.364 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:56:59.364 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:56:59.364 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T06:56:59.365 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 7 2024-02-10T06:56:59.365 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:56:59.365 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:56:59.365 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:56:59.365 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:56:59.365 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:56:59.365 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-02-10T06:56:59.365 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 11 2024-02-10T06:56:59.365 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:56:59.365 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:56:59.701 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:56:59.701 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:56:59.701 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:56:59.701 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:56:59.702 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:56:59.702 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:56:59.702 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-02-10T06:56:59.702 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-02-10T06:56:59.702 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:56:59.702 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "13/23 daemons upgraded", 2024-02-10T06:56:59.702 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T06:56:59.702 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:56:59.702 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:57:00.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:59 smithi008 ceph-mon[66415]: from='client.34442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:57:00.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:59 smithi008 ceph-mon[66415]: pgmap v263: 97 pgs: 97 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 41 KiB/s wr, 21 op/s; 2.4 MiB/s, 3 keys/s, 53 objects/s recovering 2024-02-10T06:57:00.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:59 smithi008 ceph-mon[66415]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:57:00.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:59 smithi008 ceph-mon[66415]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 86/438 objects degraded (19.635%), 3 pgs degraded) 2024-02-10T06:57:00.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:59 smithi008 ceph-mon[66415]: Cluster is now healthy 2024-02-10T06:57:00.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:59 smithi008 ceph-mon[66415]: from='client.34450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:57:00.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:56:59 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/1167518065' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:00.108 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:57:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:59 smithi029 ceph-mon[53210]: from='client.34442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:57:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:59 smithi029 ceph-mon[53210]: pgmap v263: 97 pgs: 97 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 41 KiB/s wr, 21 op/s; 2.4 MiB/s, 3 keys/s, 53 objects/s recovering 2024-02-10T06:57:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:59 smithi029 ceph-mon[53210]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:57:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:59 smithi029 ceph-mon[53210]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 86/438 objects degraded (19.635%), 3 pgs degraded) 2024-02-10T06:57:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:59 smithi029 ceph-mon[53210]: Cluster is now healthy 2024-02-10T06:57:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:59 smithi029 ceph-mon[53210]: from='client.34450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:57:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:56:59 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/1167518065' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:01.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:00 smithi008 ceph-mon[66415]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:57:01.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:00 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/679526998' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:57:01.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:00 smithi029 ceph-mon[53210]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:57:01.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:00 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/679526998' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:57:02.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:01 smithi008 ceph-mon[66415]: pgmap v264: 97 pgs: 97 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 5.5 KiB/s wr, 4 op/s; 1.9 MiB/s, 2 keys/s, 35 objects/s recovering 2024-02-10T06:57:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:01 smithi029 ceph-mon[53210]: pgmap v264: 97 pgs: 97 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 5.5 KiB/s wr, 4 op/s; 1.9 MiB/s, 2 keys/s, 35 objects/s recovering 2024-02-10T06:57:04.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:03 smithi008 ceph-mon[66415]: pgmap v265: 97 pgs: 97 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 236 B/s rd, 709 B/s wr, 3 op/s; 1.7 MiB/s, 2 keys/s, 31 objects/s recovering 2024-02-10T06:57:04.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:04.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:57:04.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:03 smithi029 ceph-mon[53210]: pgmap v265: 97 pgs: 97 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 236 B/s rd, 709 B/s wr, 3 op/s; 1.7 MiB/s, 2 keys/s, 31 objects/s recovering 2024-02-10T06:57:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:57:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:06.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:05 smithi008 ceph-mon[66415]: pgmap v266: 97 pgs: 97 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 511 B/s wr, 1 op/s; 1.5 MiB/s, 18 objects/s recovering 2024-02-10T06:57:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:05 smithi029 ceph-mon[53210]: pgmap v266: 97 pgs: 97 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 511 B/s wr, 1 op/s; 1.5 MiB/s, 18 objects/s recovering 2024-02-10T06:57:07.016 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:06 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:57:07.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:06 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:57:08.019 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:07 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:57:08.019 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:07 smithi029 ceph-mon[53210]: Upgrade: osd.7 is safe to restart 2024-02-10T06:57:08.019 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:07 smithi029 ceph-mon[53210]: pgmap v267: 97 pgs: 97 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 439 B/s wr, 1 op/s; 1.3 MiB/s, 15 objects/s recovering 2024-02-10T06:57:08.028 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:07 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T06:57:08.028 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:07 smithi008 ceph-mon[66415]: Upgrade: osd.7 is safe to restart 2024-02-10T06:57:08.028 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:07 smithi008 ceph-mon[66415]: pgmap v267: 97 pgs: 97 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 439 B/s wr, 1 op/s; 1.3 MiB/s, 15 objects/s recovering 2024-02-10T06:57:09.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:09 smithi029 ceph-mon[53210]: Upgrade: Updating osd.7 2024-02-10T06:57:09.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:09 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:09.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:09 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T06:57:09.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:09 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:57:09.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:09 smithi029 ceph-mon[53210]: Deploying daemon osd.7 on smithi029 2024-02-10T06:57:09.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:09 smithi029 ceph-mon[53210]: pgmap v268: 97 pgs: 97 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 341 B/s wr, 0 op/s; 341 KiB/s, 9 objects/s recovering 2024-02-10T06:57:09.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:09 smithi008 ceph-mon[66415]: Upgrade: Updating osd.7 2024-02-10T06:57:09.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:09 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:09.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:09 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T06:57:09.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:09 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:57:09.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:09 smithi008 ceph-mon[66415]: Deploying daemon osd.7 on smithi029 2024-02-10T06:57:09.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:09 smithi008 ceph-mon[66415]: pgmap v268: 97 pgs: 97 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 341 B/s wr, 0 op/s; 341 KiB/s, 9 objects/s recovering 2024-02-10T06:57:11.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:11 smithi029 ceph-mon[53210]: pgmap v269: 97 pgs: 97 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:57:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:11 smithi008 ceph-mon[66415]: pgmap v269: 97 pgs: 97 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:57:12.128 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:11 smithi029 systemd[1]: Stopping Ceph osd.7 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:57:12.426 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:12 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[40346]: 2024-02-10T06:57:12.161+0000 7f22f3940700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:57:12.426 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:12 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[40346]: 2024-02-10T06:57:12.161+0000 7f22f3940700 -1 osd.7 133 *** Got signal Terminated *** 2024-02-10T06:57:12.426 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:12 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[40346]: 2024-02-10T06:57:12.161+0000 7f22f3940700 -1 osd.7 133 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:57:12.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:12 smithi029 ceph-mon[53210]: osd.7 marked itself down and dead 2024-02-10T06:57:12.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:12 smithi008 ceph-mon[66415]: osd.7 marked itself down and dead 2024-02-10T06:57:13.306 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:12 smithi029 podman[64461]: 2024-02-10 06:57:12.995462936 +0000 UTC m=+0.952797996 container died badc7a3ebb0cf35eb96e2921e9b87250b6f27dd358d3a07e1c072d2e6f9e79b4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, build-date=2022-05-03T08:36:31.336870, vcs-type=git, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64) 2024-02-10T06:57:13.668 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:13 smithi029 ceph-mon[53210]: pgmap v270: 97 pgs: 97 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:57:13.668 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:13 smithi029 ceph-mon[53210]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:57:13.668 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:13 smithi029 ceph-mon[53210]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-10T06:57:13.668 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:13 smithi029 ceph-mon[53210]: osdmap e134: 8 total, 7 up, 8 in 2024-02-10T06:57:13.668 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:13 smithi029 podman[64461]: 2024-02-10 06:57:13.405653643 +0000 UTC m=+1.362988704 container cleanup badc7a3ebb0cf35eb96e2921e9b87250b6f27dd358d3a07e1c072d2e6f9e79b4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, vendor=Red Hat, Inc., architecture=x86_64, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, name=centos-stream, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, version=8, GIT_BRANCH=HEAD, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, distribution-scope=public) 2024-02-10T06:57:13.669 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:13 smithi029 bash[64461]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7 2024-02-10T06:57:13.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:13 smithi008 ceph-mon[66415]: pgmap v270: 97 pgs: 97 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:57:13.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:13 smithi008 ceph-mon[66415]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T06:57:13.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:13 smithi008 ceph-mon[66415]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-10T06:57:13.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:13 smithi008 ceph-mon[66415]: osdmap e134: 8 total, 7 up, 8 in 2024-02-10T06:57:13.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:13 smithi029 podman[64488]: 2024-02-10 06:57:13.668188538 +0000 UTC m=+0.669078788 container remove badc7a3ebb0cf35eb96e2921e9b87250b6f27dd358d3a07e1c072d2e6f9e79b4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, GIT_CLEAN=True, distribution-scope=public, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , name=centos-stream, ceph=True, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, RELEASE=HEAD) 2024-02-10T06:57:14.344 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:14 smithi029 podman[64581]: 2024-02-10 06:57:13.996383381 +0000 UTC m=+0.017824985 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:57:14.344 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:14 smithi029 podman[64581]: 2024-02-10 06:57:14.120200212 +0000 UTC m=+0.141641802 container create b8efb9a47d0465cfa7e14fbaecb5ef3a8848da6bed661054a478705f25096cb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, RELEASE=main-290212a, CEPH_POINT_RELEASE=) 2024-02-10T06:57:14.344 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:14 smithi029 podman[64581]: 2024-02-10 06:57:14.341177529 +0000 UTC m=+0.362619122 container init b8efb9a47d0465cfa7e14fbaecb5ef3a8848da6bed661054a478705f25096cb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131) 2024-02-10T06:57:14.645 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:14 smithi029 ceph-mon[53210]: osdmap e135: 8 total, 7 up, 8 in 2024-02-10T06:57:14.645 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:14 smithi029 podman[64581]: 2024-02-10 06:57:14.344169039 +0000 UTC m=+0.365610627 container start b8efb9a47d0465cfa7e14fbaecb5ef3a8848da6bed661054a478705f25096cb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, RELEASE=main-290212a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-10T06:57:14.645 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:14 smithi029 podman[64581]: 2024-02-10 06:57:14.468283289 +0000 UTC m=+0.489724900 container attach b8efb9a47d0465cfa7e14fbaecb5ef3a8848da6bed661054a478705f25096cb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-290212a, ceph=True, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:57:14.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:14 smithi008 ceph-mon[66415]: osdmap e135: 8 total, 7 up, 8 in 2024-02-10T06:57:14.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:14 smithi029 podman[64581]: 2024-02-10 06:57:14.644957477 +0000 UTC m=+0.666399070 container died b8efb9a47d0465cfa7e14fbaecb5ef3a8848da6bed661054a478705f25096cb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-10T06:57:15.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:15 smithi029 ceph-mon[53210]: pgmap v273: 97 pgs: 3 active+undersized+degraded, 17 peering, 5 stale+active+clean, 72 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 7/435 objects degraded (1.609%) 2024-02-10T06:57:15.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:15 smithi029 ceph-mon[53210]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-02-10T06:57:15.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:15 smithi029 ceph-mon[53210]: Health check failed: Degraded data redundancy: 7/435 objects degraded (1.609%), 3 pgs degraded (PG_DEGRADED) 2024-02-10T06:57:15.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:15 smithi029 podman[64607]: 2024-02-10 06:57:15.266767441 +0000 UTC m=+0.612706380 container remove b8efb9a47d0465cfa7e14fbaecb5ef3a8848da6bed661054a478705f25096cb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-02-10T06:57:15.709 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:15 smithi029 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.7.service: Deactivated successfully. 2024-02-10T06:57:15.709 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:15 smithi029 systemd[1]: Stopped Ceph osd.7 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:57:15.709 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:15 smithi029 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.7.service: Consumed 8min 9.076s CPU time. 2024-02-10T06:57:15.709 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:15 smithi029 systemd[1]: Starting Ceph osd.7 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T06:57:15.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:15 smithi008 ceph-mon[66415]: pgmap v273: 97 pgs: 3 active+undersized+degraded, 17 peering, 5 stale+active+clean, 72 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 7/435 objects degraded (1.609%) 2024-02-10T06:57:15.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:15 smithi008 ceph-mon[66415]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-02-10T06:57:15.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:15 smithi008 ceph-mon[66415]: Health check failed: Degraded data redundancy: 7/435 objects degraded (1.609%), 3 pgs degraded (PG_DEGRADED) 2024-02-10T06:57:16.050 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:15 smithi029 podman[64709]: 2024-02-10 06:57:15.619215175 +0000 UTC m=+0.018027671 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:57:16.050 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:15 smithi029 podman[64709]: 2024-02-10 06:57:15.762340236 +0000 UTC m=+0.161152767 container create 4e0278004c5e23a6de3568fd2d1430dc630a2801e92ee08a74361c573638886c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_BRANCH=HEAD) 2024-02-10T06:57:16.050 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 podman[64709]: 2024-02-10 06:57:16.050315398 +0000 UTC m=+0.449127890 container init 4e0278004c5e23a6de3568fd2d1430dc630a2801e92ee08a74361c573638886c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-activate, RELEASE=main-290212a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, ceph=True) 2024-02-10T06:57:16.458 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 podman[64709]: 2024-02-10 06:57:16.053638866 +0000 UTC m=+0.452451363 container start 4e0278004c5e23a6de3568fd2d1430dc630a2801e92ee08a74361c573638886c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-activate, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-10T06:57:16.458 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 podman[64709]: 2024-02-10 06:57:16.17581375 +0000 UTC m=+0.574626251 container attach 4e0278004c5e23a6de3568fd2d1430dc630a2801e92ee08a74361c573638886c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-activate, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-290212a, io.buildah.version=1.33.2) 2024-02-10T06:57:16.823 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-activate[64724]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T06:57:16.823 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-activate[64724]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T06:57:16.823 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 bash[64709]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T06:57:16.823 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 bash[64709]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T06:57:16.824 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 bash[64709]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T06:57:16.824 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 bash[64709]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T06:57:16.824 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 bash[64709]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-10T06:57:16.824 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 bash[64709]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T06:57:16.824 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 bash[64709]: --> ceph-volume raw activate successful for osd ID: 7 2024-02-10T06:57:16.824 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-activate[64724]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T06:57:16.824 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-activate[64724]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T06:57:16.824 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-activate[64724]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-10T06:57:16.824 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-activate[64724]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T06:57:16.824 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-activate[64724]: --> ceph-volume raw activate successful for osd ID: 7 2024-02-10T06:57:16.824 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:16 smithi029 podman[64709]: 2024-02-10 06:57:16.525192396 +0000 UTC m=+0.924004878 container died 4e0278004c5e23a6de3568fd2d1430dc630a2801e92ee08a74361c573638886c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-10T06:57:17.433 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:17 smithi029 podman[64841]: 2024-02-10 06:57:17.176007905 +0000 UTC m=+0.646036502 container remove 4e0278004c5e23a6de3568fd2d1430dc630a2801e92ee08a74361c573638886c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-10T06:57:17.433 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:17 smithi029 podman[64906]: 2024-02-10 06:57:17.297014504 +0000 UTC m=+0.018392312 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:57:17.433 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:17 smithi029 podman[64906]: 2024-02-10 06:57:17.411020298 +0000 UTC m=+0.132398140 container create 156ce7027d7a10da781c50003137af43f916f588bee528a7b927289d5be4da15 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD) 2024-02-10T06:57:17.698 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:17 smithi029 ceph-mon[53210]: pgmap v274: 97 pgs: 5 active+undersized, 12 active+undersized+degraded, 17 peering, 1 stale+active+clean, 62 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 43/435 objects degraded (9.885%) 2024-02-10T06:57:17.699 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:17 smithi029 podman[64906]: 2024-02-10 06:57:17.675906049 +0000 UTC m=+0.397283882 container init 156ce7027d7a10da781c50003137af43f916f588bee528a7b927289d5be4da15 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-10T06:57:17.699 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:17 smithi029 podman[64906]: 2024-02-10 06:57:17.679091265 +0000 UTC m=+0.400469072 container start 156ce7027d7a10da781c50003137af43f916f588bee528a7b927289d5be4da15 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_CLEAN=True) 2024-02-10T06:57:17.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:17 smithi008 ceph-mon[66415]: pgmap v274: 97 pgs: 5 active+undersized, 12 active+undersized+degraded, 17 peering, 1 stale+active+clean, 62 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 43/435 objects degraded (9.885%) 2024-02-10T06:57:17.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:17 smithi029 ceph-osd[64925]: -- 172.21.15.29:0/2946875902 <== mon.1 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561369b451e0 con 0x561369b17000 2024-02-10T06:57:17.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:17 smithi029 bash[64906]: 156ce7027d7a10da781c50003137af43f916f588bee528a7b927289d5be4da15 2024-02-10T06:57:17.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:17 smithi029 systemd[1]: Started Ceph osd.7 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T06:57:18.707 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:18 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T06:57:18.225+0000 7f829468c7c0 -1 Falling back to public interface 2024-02-10T06:57:18.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:18.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:18.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:57:18.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:18.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:57:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:57:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:57:20.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:19 smithi029 ceph-mon[53210]: pgmap v275: 97 pgs: 20 active+undersized, 22 active+undersized+degraded, 55 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 82/435 objects degraded (18.851%) 2024-02-10T06:57:20.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:19 smithi029 ceph-mon[53210]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-02-10T06:57:20.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:19 smithi008 ceph-mon[66415]: pgmap v275: 97 pgs: 20 active+undersized, 22 active+undersized+degraded, 55 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 82/435 objects degraded (18.851%) 2024-02-10T06:57:20.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:19 smithi008 ceph-mon[66415]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-02-10T06:57:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:21 smithi029 ceph-mon[53210]: pgmap v276: 97 pgs: 20 active+undersized, 22 active+undersized+degraded, 55 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 82/435 objects degraded (18.851%) 2024-02-10T06:57:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:21 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:21 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:22.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:21 smithi008 ceph-mon[66415]: pgmap v276: 97 pgs: 20 active+undersized, 22 active+undersized+degraded, 55 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 82/435 objects degraded (18.851%) 2024-02-10T06:57:22.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:21 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:22.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:21 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:23 smithi029 ceph-mon[53210]: pgmap v277: 97 pgs: 20 active+undersized, 22 active+undersized+degraded, 55 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 82/435 objects degraded (18.851%) 2024-02-10T06:57:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:23 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:23 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:23 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 82/435 objects degraded (18.851%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T06:57:24.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:23 smithi008 ceph-mon[66415]: pgmap v277: 97 pgs: 20 active+undersized, 22 active+undersized+degraded, 55 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 82/435 objects degraded (18.851%) 2024-02-10T06:57:24.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:23 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:24.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:23 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:24.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:23 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 82/435 objects degraded (18.851%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T06:57:25.159 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:24 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T06:57:24.950+0000 7f829468c7c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-02-10T06:57:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:25 smithi029 ceph-mon[53210]: pgmap v278: 97 pgs: 20 active+undersized, 22 active+undersized+degraded, 55 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 82/435 objects degraded (18.851%) 2024-02-10T06:57:26.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:25 smithi008 ceph-mon[66415]: pgmap v278: 97 pgs: 20 active+undersized, 22 active+undersized+degraded, 55 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 82/435 objects degraded (18.851%) 2024-02-10T06:57:26.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:26 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T06:57:26.658+0000 7f829468c7c0 -1 osd.7 133 log_to_monitors true 2024-02-10T06:57:27.988 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: pgmap v279: 97 pgs: 20 active+undersized, 22 active+undersized+degraded, 55 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 82/435 objects degraded (18.851%) 2024-02-10T06:57:27.988 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:27.988 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='osd.7 [v2:172.21.15.29:6824/2287949117,v1:172.21.15.29:6825/2287949117]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T06:57:27.988 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T06:57:27.988 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:27.989 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:57:27.989 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:57:27.989 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:27.989 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:27.989 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:57:27.989 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:27.989 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:27.989 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:27.989 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:27.989 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: Upgrade: Setting container_image for all osd 2024-02-10T06:57:27.989 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:27.989 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-10T06:57:27.989 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-10T06:57:27.989 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-10T06:57:27.990 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-10T06:57:27.990 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-10T06:57:27.990 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-10T06:57:27.990 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-10T06:57:27.990 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-10T06:57:27.990 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-10T06:57:27.990 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-10T06:57:27.990 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-10T06:57:27.990 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-10T06:57:27.990 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-10T06:57:27.990 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-10T06:57:27.990 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-10T06:57:27.990 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-10T06:57:27.990 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: Upgrade: Setting require_osd_release to 19 squid 2024-02-10T06:57:27.991 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-10T06:57:28.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: pgmap v279: 97 pgs: 20 active+undersized, 22 active+undersized+degraded, 55 active+clean; 234 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 82/435 objects degraded (18.851%) 2024-02-10T06:57:28.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:28.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='osd.7 [v2:172.21.15.29:6824/2287949117,v1:172.21.15.29:6825/2287949117]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T06:57:28.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T06:57:28.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:28.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:57:28.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:57:28.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:28.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:28.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:57:28.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:28.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:28.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:28.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:28.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: Upgrade: Setting container_image for all osd 2024-02-10T06:57:28.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:28.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-10T06:57:28.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-10T06:57:28.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-10T06:57:28.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-10T06:57:28.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-10T06:57:28.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-10T06:57:28.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-10T06:57:28.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-10T06:57:28.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-10T06:57:28.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-10T06:57:28.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-10T06:57:28.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-10T06:57:28.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-10T06:57:28.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-10T06:57:28.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-10T06:57:28.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-10T06:57:28.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: Upgrade: Setting require_osd_release to 19 squid 2024-02-10T06:57:28.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-10T06:57:28.458 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 06:57:27 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T06:57:27.986+0000 7f8281adc700 -1 osd.7 133 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:57:29.011 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:28 smithi029 ceph-mon[53210]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-10T06:57:29.012 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:28 smithi029 ceph-mon[53210]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T06:57:29.012 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:28 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-10T06:57:29.012 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:28 smithi029 ceph-mon[53210]: osdmap e136: 8 total, 7 up, 8 in 2024-02-10T06:57:29.012 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:28 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi008.vfwcte"]}]: dispatch 2024-02-10T06:57:29.012 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:28 smithi029 ceph-mon[53210]: Upgrade: It appears safe to stop mds.foofs.smithi008.vfwcte 2024-02-10T06:57:29.012 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:28 smithi029 ceph-mon[53210]: from='osd.7 [v2:172.21.15.29:6824/2287949117,v1:172.21.15.29:6825/2287949117]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:57:29.012 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:28 smithi029 ceph-mon[53210]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:57:29.012 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:28 smithi029 ceph-mon[53210]: from='osd.7 ' entity='osd.7' 2024-02-10T06:57:29.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:28 smithi008 ceph-mon[66415]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-10T06:57:29.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:28 smithi008 ceph-mon[66415]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T06:57:29.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:28 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-10T06:57:29.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:28 smithi008 ceph-mon[66415]: osdmap e136: 8 total, 7 up, 8 in 2024-02-10T06:57:29.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:28 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi008.vfwcte"]}]: dispatch 2024-02-10T06:57:29.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:28 smithi008 ceph-mon[66415]: Upgrade: It appears safe to stop mds.foofs.smithi008.vfwcte 2024-02-10T06:57:29.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:28 smithi008 ceph-mon[66415]: from='osd.7 [v2:172.21.15.29:6824/2287949117,v1:172.21.15.29:6825/2287949117]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:57:29.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:28 smithi008 ceph-mon[66415]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T06:57:29.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:28 smithi008 ceph-mon[66415]: from='osd.7 ' entity='osd.7' 2024-02-10T06:57:30.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:29 smithi008 ceph-mon[66415]: pgmap v281: 97 pgs: 20 active+undersized, 22 active+undersized+degraded, 55 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 204 B/s rd, 0 op/s; 82/435 objects degraded (18.851%) 2024-02-10T06:57:30.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:29 smithi008 ceph-mon[66415]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:57:30.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:29 smithi008 ceph-mon[66415]: osd.7 [v2:172.21.15.29:6824/2287949117,v1:172.21.15.29:6825/2287949117] boot 2024-02-10T06:57:30.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:29 smithi008 ceph-mon[66415]: osdmap e137: 8 total, 8 up, 8 in 2024-02-10T06:57:30.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:29 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:57:30.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:29 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:30.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:29 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.vfwcte", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:57:30.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:29 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:57:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:29 smithi029 ceph-mon[53210]: pgmap v281: 97 pgs: 20 active+undersized, 22 active+undersized+degraded, 55 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 204 B/s rd, 0 op/s; 82/435 objects degraded (18.851%) 2024-02-10T06:57:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:29 smithi029 ceph-mon[53210]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T06:57:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:29 smithi029 ceph-mon[53210]: osd.7 [v2:172.21.15.29:6824/2287949117,v1:172.21.15.29:6825/2287949117] boot 2024-02-10T06:57:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:29 smithi029 ceph-mon[53210]: osdmap e137: 8 total, 8 up, 8 in 2024-02-10T06:57:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:29 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T06:57:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:29 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:29 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.vfwcte", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:57:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:29 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:57:30.476 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:57:31.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:30 smithi008 ceph-mon[66415]: Upgrade: Updating mds.foofs.smithi008.vfwcte 2024-02-10T06:57:31.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:30 smithi008 ceph-mon[66415]: Deploying daemon mds.foofs.smithi008.vfwcte on smithi008 2024-02-10T06:57:31.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:30 smithi008 ceph-mon[66415]: osdmap e138: 8 total, 8 up, 8 in 2024-02-10T06:57:31.159 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:57:31.160 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (9m) 2m ago 19m 15.6M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:57:31.160 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (5m) 2m ago 19m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T06:57:31.160 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (5m) 9s ago 18m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T06:57:31.160 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (9m) 2m ago 19m 43.0M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:57:31.160 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (13m) 2m ago 13m 38.5M - 17.2.0 e1d6a67b021e d6ae62325d52 2024-02-10T06:57:31.160 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (13m) 9s ago 13m 15.8M - 17.2.0 e1d6a67b021e 7001f288148a 2024-02-10T06:57:31.160 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (8m) 2m ago 20m 507M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:57:31.160 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (7m) 9s ago 18m 452M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:57:31.160 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (7m) 2m ago 21m 53.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:57:31.160 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (6m) 9s ago 18m 45.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:57:31.160 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (13m) 2m ago 13m 271M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:57:31.160 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (12m) 9s ago 12m 84.7M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:57:31.160 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (9m) 2m ago 19m 11.4M - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:57:31.161 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (8m) 9s ago 18m 17.6M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:57:31.161 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (5m) 2m ago 18m 355M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 acb0107eae6c 2024-02-10T06:57:31.161 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (4m) 2m ago 17m 282M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 09363f1c35ae 2024-02-10T06:57:31.161 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (3m) 2m ago 17m 237M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 bd6bbb022085 2024-02-10T06:57:31.161 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (2m) 2m ago 16m 15.0M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc3683d167e9 2024-02-10T06:57:31.161 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (2m) 9s ago 16m 344M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 2b1f7cbc27ce 2024-02-10T06:57:31.161 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (92s) 9s ago 15m 306M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 ea4052b4cafa 2024-02-10T06:57:31.161 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (50s) 9s ago 15m 258M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 d89cf7ba8295 2024-02-10T06:57:31.161 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (13s) 9s ago 14m 12.6M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 156ce7027d7a 2024-02-10T06:57:31.161 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (7m) 2m ago 18m 63.3M - 2.33.4 514e6a882f6e 193819b86dea 2024-02-10T06:57:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:30 smithi029 ceph-mon[53210]: Upgrade: Updating mds.foofs.smithi008.vfwcte 2024-02-10T06:57:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:30 smithi029 ceph-mon[53210]: Deploying daemon mds.foofs.smithi008.vfwcte on smithi008 2024-02-10T06:57:31.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:30 smithi029 ceph-mon[53210]: osdmap e138: 8 total, 8 up, 8 in 2024-02-10T06:57:31.623 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:57:31.623 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:57:31.623 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:57:31.623 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:57:31.624 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:57:31.624 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:57:31.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:57:31.624 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:57:31.624 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T06:57:31.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:57:31.624 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:57:31.624 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T06:57:31.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:57:31.624 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:57:31.624 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-10T06:57:31.624 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 12 2024-02-10T06:57:31.624 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:57:31.624 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:57:31.985 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:57:31.986 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:57:31.986 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:57:31.986 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:57:31.986 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:57:31.986 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:57:31.986 INFO:teuthology.orchestra.run.smithi008.stdout: "osd", 2024-02-10T06:57:31.986 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-02-10T06:57:31.986 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-02-10T06:57:31.986 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:57:31.986 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "14/23 daemons upgraded", 2024-02-10T06:57:31.986 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading mds daemons", 2024-02-10T06:57:31.986 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:57:31.987 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:57:32.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:31 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008[66411]: 2024-02-10T06:57:31.767+0000 7f8c8c43c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T06:57:32.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:31 smithi008 ceph-mon[66415]: pgmap v284: 97 pgs: 10 peering, 15 active+undersized, 17 active+undersized+degraded, 55 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 64/435 objects degraded (14.713%) 2024-02-10T06:57:32.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:31 smithi008 ceph-mon[66415]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:57:32.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:31 smithi008 ceph-mon[66415]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:57:32.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:31 smithi008 ceph-mon[66415]: Health check update: Degraded data redundancy: 64/435 objects degraded (14.713%), 17 pgs degraded (PG_DEGRADED) 2024-02-10T06:57:32.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:31 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/809449767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:31 smithi008 ceph-mon[66415]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-10T06:57:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:31 smithi008 ceph-mon[66415]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T06:57:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:31 smithi008 ceph-mon[66415]: osdmap e139: 8 total, 8 up, 8 in 2024-02-10T06:57:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:31 smithi008 ceph-mon[66415]: Standby daemon mds.foofs.smithi029.bwmcyt assigned to filesystem foofs as rank 0 2024-02-10T06:57:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:31 smithi008 ceph-mon[66415]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-10T06:57:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:31 smithi008 ceph-mon[66415]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-10T06:57:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:32 smithi008 ceph-mon[66415]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-02-10T06:57:32.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:32 smithi008 ceph-mon[66415]: fsmap foofs:1/1 {0=foofs.smithi029.bwmcyt=up:replay} 2024-02-10T06:57:32.397 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available; Degraded data redundancy: 64/435 objects degraded (14.713%), 17 pgs degraded 2024-02-10T06:57:32.397 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-10T06:57:32.398 INFO:teuthology.orchestra.run.smithi008.stdout: fs foofs is degraded 2024-02-10T06:57:32.398 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available 2024-02-10T06:57:32.398 INFO:teuthology.orchestra.run.smithi008.stdout: have 0; want 1 more 2024-02-10T06:57:32.398 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 64/435 objects degraded (14.713%), 17 pgs degraded 2024-02-10T06:57:32.398 INFO:teuthology.orchestra.run.smithi008.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-02-10T06:57:32.398 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-02-10T06:57:32.398 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-02-10T06:57:32.398 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-02-10T06:57:32.398 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.8 is active+undersized+degraded, acting [0,1] 2024-02-10T06:57:32.398 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-02-10T06:57:32.398 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-02-10T06:57:32.399 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-02-10T06:57:32.399 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-02-10T06:57:32.399 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-02-10T06:57:32.399 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-02-10T06:57:32.399 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.18 is active+undersized+degraded, acting [5,4] 2024-02-10T06:57:32.399 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1b is active+undersized+degraded, acting [3,6] 2024-02-10T06:57:32.399 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-02-10T06:57:32.399 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1d is active+undersized+degraded, acting [6,0] 2024-02-10T06:57:32.399 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-02-10T06:57:32.399 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-02-10T06:57:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:31 smithi029 ceph-mon[53210]: pgmap v284: 97 pgs: 10 peering, 15 active+undersized, 17 active+undersized+degraded, 55 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 64/435 objects degraded (14.713%) 2024-02-10T06:57:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:31 smithi029 ceph-mon[53210]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:57:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:31 smithi029 ceph-mon[53210]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:57:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:31 smithi029 ceph-mon[53210]: Health check update: Degraded data redundancy: 64/435 objects degraded (14.713%), 17 pgs degraded (PG_DEGRADED) 2024-02-10T06:57:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:31 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/809449767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:31 smithi029 ceph-mon[53210]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-10T06:57:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:31 smithi029 ceph-mon[53210]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T06:57:32.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:31 smithi029 ceph-mon[53210]: osdmap e139: 8 total, 8 up, 8 in 2024-02-10T06:57:32.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:31 smithi029 ceph-mon[53210]: Standby daemon mds.foofs.smithi029.bwmcyt assigned to filesystem foofs as rank 0 2024-02-10T06:57:32.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:31 smithi029 ceph-mon[53210]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-10T06:57:32.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:31 smithi029 ceph-mon[53210]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-10T06:57:32.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:31 smithi029 ceph-mon[53210]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-02-10T06:57:32.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:31 smithi029 ceph-mon[53210]: fsmap foofs:1/1 {0=foofs.smithi029.bwmcyt=up:replay} 2024-02-10T06:57:33.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:33 smithi008 ceph-mon[66415]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:57:33.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:33 smithi008 ceph-mon[66415]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:57:33.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:33 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/4087902239' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:57:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:33 smithi029 ceph-mon[53210]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:57:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:33 smithi029 ceph-mon[53210]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:57:33.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:33 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/4087902239' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:57:34.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:34 smithi008 ceph-mon[66415]: pgmap v286: 97 pgs: 10 peering, 11 active+undersized, 14 active+undersized+degraded, 62 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 53/435 objects degraded (12.184%) 2024-02-10T06:57:34.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:34 smithi008 ceph-mon[66415]: reconnect by client.24363 172.21.15.29:0/3804362279 after 0.000999984 2024-02-10T06:57:34.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:34 smithi008 ceph-mon[66415]: reconnect by client.14558 172.21.15.8:0/1464178136 after 0.000999984 2024-02-10T06:57:34.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:34 smithi008 ceph-mon[66415]: mds.? [v2:172.21.15.29:6832/1338152565,v1:172.21.15.29:6833/1338152565] up:reconnect 2024-02-10T06:57:34.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:34 smithi008 ceph-mon[66415]: fsmap foofs:1/1 {0=foofs.smithi029.bwmcyt=up:reconnect} 2024-02-10T06:57:34.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:34 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:34.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:34 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:57:34.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:34 smithi029 ceph-mon[53210]: pgmap v286: 97 pgs: 10 peering, 11 active+undersized, 14 active+undersized+degraded, 62 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 53/435 objects degraded (12.184%) 2024-02-10T06:57:34.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:34 smithi029 ceph-mon[53210]: reconnect by client.24363 172.21.15.29:0/3804362279 after 0.000999984 2024-02-10T06:57:34.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:34 smithi029 ceph-mon[53210]: reconnect by client.14558 172.21.15.8:0/1464178136 after 0.000999984 2024-02-10T06:57:34.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:34 smithi029 ceph-mon[53210]: mds.? [v2:172.21.15.29:6832/1338152565,v1:172.21.15.29:6833/1338152565] up:reconnect 2024-02-10T06:57:34.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:34 smithi029 ceph-mon[53210]: fsmap foofs:1/1 {0=foofs.smithi029.bwmcyt=up:reconnect} 2024-02-10T06:57:34.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:34 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:34.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:34 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:57:35.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:35 smithi008 ceph-mon[66415]: mds.? [v2:172.21.15.29:6832/1338152565,v1:172.21.15.29:6833/1338152565] up:rejoin 2024-02-10T06:57:35.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:35 smithi008 ceph-mon[66415]: fsmap foofs:1/1 {0=foofs.smithi029.bwmcyt=up:rejoin} 2024-02-10T06:57:35.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:35 smithi008 ceph-mon[66415]: daemon mds.foofs.smithi029.bwmcyt is now active in filesystem foofs as rank 0 2024-02-10T06:57:35.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:35 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.vfwcte"}]: dispatch 2024-02-10T06:57:35.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:35 smithi008 ceph-mon[66415]: pgmap v287: 97 pgs: 10 peering, 87 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 6 op/s 2024-02-10T06:57:35.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:35 smithi029 ceph-mon[53210]: mds.? [v2:172.21.15.29:6832/1338152565,v1:172.21.15.29:6833/1338152565] up:rejoin 2024-02-10T06:57:35.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:35 smithi029 ceph-mon[53210]: fsmap foofs:1/1 {0=foofs.smithi029.bwmcyt=up:rejoin} 2024-02-10T06:57:35.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:35 smithi029 ceph-mon[53210]: daemon mds.foofs.smithi029.bwmcyt is now active in filesystem foofs as rank 0 2024-02-10T06:57:35.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:35 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.vfwcte"}]: dispatch 2024-02-10T06:57:35.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:35 smithi029 ceph-mon[53210]: pgmap v287: 97 pgs: 10 peering, 87 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 6 op/s 2024-02-10T06:57:36.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:36 smithi008 ceph-mon[66415]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/435 objects degraded (12.184%), 14 pgs degraded) 2024-02-10T06:57:36.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:36 smithi008 ceph-mon[66415]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-10T06:57:36.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:36 smithi008 ceph-mon[66415]: mds.? [v2:172.21.15.29:6832/1338152565,v1:172.21.15.29:6833/1338152565] up:active 2024-02-10T06:57:36.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:36 smithi008 ceph-mon[66415]: fsmap foofs:1 {0=foofs.smithi029.bwmcyt=up:active} 2024-02-10T06:57:36.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:36 smithi029 ceph-mon[53210]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/435 objects degraded (12.184%), 14 pgs degraded) 2024-02-10T06:57:36.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:36 smithi029 ceph-mon[53210]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-10T06:57:36.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:36 smithi029 ceph-mon[53210]: mds.? [v2:172.21.15.29:6832/1338152565,v1:172.21.15.29:6833/1338152565] up:active 2024-02-10T06:57:36.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:36 smithi029 ceph-mon[53210]: fsmap foofs:1 {0=foofs.smithi029.bwmcyt=up:active} 2024-02-10T06:57:37.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:37 smithi008 ceph-mon[66415]: pgmap v288: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 26 MiB/s rd, 138 B/s wr, 8 op/s 2024-02-10T06:57:37.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:37 smithi029 ceph-mon[53210]: pgmap v288: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 26 MiB/s rd, 138 B/s wr, 8 op/s 2024-02-10T06:57:40.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:39 smithi008 ceph-mon[66415]: pgmap v289: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 244 B/s wr, 11 op/s 2024-02-10T06:57:40.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:39 smithi029 ceph-mon[53210]: pgmap v289: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 244 B/s wr, 11 op/s 2024-02-10T06:57:42.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:41 smithi008 ceph-mon[66415]: pgmap v290: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 204 B/s wr, 11 op/s 2024-02-10T06:57:42.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:41 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:42.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:41 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:42.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:41 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:57:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:41 smithi029 ceph-mon[53210]: pgmap v290: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 204 B/s wr, 11 op/s 2024-02-10T06:57:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:41 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:41 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:41 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:57:43.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:42 smithi008 ceph-mon[66415]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-10T06:57:43.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:42 smithi008 ceph-mon[66415]: Cluster is now healthy 2024-02-10T06:57:43.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:42 smithi008 ceph-mon[66415]: mds.? [v2:172.21.15.8:6834/2814645351,v1:172.21.15.8:6835/2814645351] up:boot 2024-02-10T06:57:43.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:42 smithi008 ceph-mon[66415]: fsmap foofs:1 {0=foofs.smithi029.bwmcyt=up:active} 1 up:standby 2024-02-10T06:57:43.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:42 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.vfwcte"}]: dispatch 2024-02-10T06:57:43.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:42 smithi029 ceph-mon[53210]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-10T06:57:43.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:42 smithi029 ceph-mon[53210]: Cluster is now healthy 2024-02-10T06:57:43.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:42 smithi029 ceph-mon[53210]: mds.? [v2:172.21.15.8:6834/2814645351,v1:172.21.15.8:6835/2814645351] up:boot 2024-02-10T06:57:43.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:42 smithi029 ceph-mon[53210]: fsmap foofs:1 {0=foofs.smithi029.bwmcyt=up:active} 1 up:standby 2024-02-10T06:57:43.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:42 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.vfwcte"}]: dispatch 2024-02-10T06:57:44.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:43 smithi008 ceph-mon[66415]: pgmap v291: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 193 B/s wr, 10 op/s 2024-02-10T06:57:44.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:43 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:44.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:43 smithi029 ceph-mon[53210]: pgmap v291: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 193 B/s wr, 10 op/s 2024-02-10T06:57:44.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:43 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:46.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:45 smithi008 ceph-mon[66415]: pgmap v292: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-02-10T06:57:46.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:45 smithi029 ceph-mon[53210]: pgmap v292: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-02-10T06:57:47.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:47.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:47.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:47 smithi029 ceph-mon[53210]: pgmap v293: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 7.2 KiB/s wr, 8 op/s 2024-02-10T06:57:47.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:47.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:47.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:47 smithi008 ceph-mon[66415]: pgmap v293: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 7.2 KiB/s wr, 8 op/s 2024-02-10T06:57:48.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:48.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:48.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:48.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:57:48.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:48.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:48.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:48.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:57:49.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:49 smithi008 ceph-mon[66415]: pgmap v294: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 7.1 KiB/s wr, 7 op/s 2024-02-10T06:57:50.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:49 smithi029 ceph-mon[53210]: pgmap v294: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 7.1 KiB/s wr, 7 op/s 2024-02-10T06:57:52.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:51 smithi029 ceph-mon[53210]: pgmap v295: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 7.0 KiB/s wr, 4 op/s 2024-02-10T06:57:52.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:52.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:52.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:57:52.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:57:52.017 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:52.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:52.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:57:52.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:52.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:52.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:52.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:52.018 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi029.bwmcyt"]}]: dispatch 2024-02-10T06:57:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:51 smithi008 ceph-mon[66415]: pgmap v295: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 7.0 KiB/s wr, 4 op/s 2024-02-10T06:57:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:57:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:57:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:57:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:52.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:52.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:57:52.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi029.bwmcyt"]}]: dispatch 2024-02-10T06:57:53.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:52 smithi008 ceph-mon[66415]: Detected new or changed devices on smithi008 2024-02-10T06:57:53.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:52 smithi008 ceph-mon[66415]: Upgrade: It appears safe to stop mds.foofs.smithi029.bwmcyt 2024-02-10T06:57:53.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:52 smithi029 ceph-mon[53210]: Detected new or changed devices on smithi008 2024-02-10T06:57:53.069 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:52 smithi029 ceph-mon[53210]: Upgrade: It appears safe to stop mds.foofs.smithi029.bwmcyt 2024-02-10T06:57:54.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:53 smithi008 ceph-mon[66415]: pgmap v296: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 7.0 KiB/s wr, 3 op/s 2024-02-10T06:57:54.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:53 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:54.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:53 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi029.bwmcyt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:57:54.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:53 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:57:54.049 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:53 smithi029 ceph-mon[53210]: pgmap v296: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 7.0 KiB/s wr, 3 op/s 2024-02-10T06:57:54.049 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:53 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:57:54.049 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:53 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi029.bwmcyt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:57:54.049 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:53 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:57:54.912 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:54 smithi029 ceph-mon[53210]: Upgrade: Updating mds.foofs.smithi029.bwmcyt 2024-02-10T06:57:54.912 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:54 smithi029 ceph-mon[53210]: Deploying daemon mds.foofs.smithi029.bwmcyt on smithi029 2024-02-10T06:57:55.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:54 smithi008 ceph-mon[66415]: Upgrade: Updating mds.foofs.smithi029.bwmcyt 2024-02-10T06:57:55.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:54 smithi008 ceph-mon[66415]: Deploying daemon mds.foofs.smithi029.bwmcyt on smithi029 2024-02-10T06:57:55.775 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:55 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008[66411]: 2024-02-10T06:57:55.507+0000 7f8c8c43c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T06:57:55.776 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:57:55 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1[75212]: 2024-02-10T06:57:55.608+0000 7f8084b6b700 -1 bdev(0x5633c72bc000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-02-10T06:57:55.776 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:57:55 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1[75212]: 2024-02-10T06:57:55.608+0000 7f808636e700 -1 bdev(0x5633c72bc000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 5 2024-02-10T06:57:55.776 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 06:57:55 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1[75212]: 2024-02-10T06:57:55.608+0000 7f8088b73700 -1 bdev(0x5633c72bc000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 7 2024-02-10T06:57:56.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:55 smithi008 ceph-mon[66415]: pgmap v297: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 7.2 KiB/s wr, 3 op/s 2024-02-10T06:57:56.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:55 smithi008 ceph-mon[66415]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-10T06:57:56.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:55 smithi008 ceph-mon[66415]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T06:57:56.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:55 smithi008 ceph-mon[66415]: osdmap e140: 8 total, 8 up, 8 in 2024-02-10T06:57:56.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:55 smithi008 ceph-mon[66415]: Standby daemon mds.foofs.smithi008.vfwcte assigned to filesystem foofs as rank 0 2024-02-10T06:57:56.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:55 smithi008 ceph-mon[66415]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-10T06:57:56.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:55 smithi008 ceph-mon[66415]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-10T06:57:56.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:55 smithi008 ceph-mon[66415]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-02-10T06:57:56.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:55 smithi008 ceph-mon[66415]: fsmap foofs:1/1 {0=foofs.smithi008.vfwcte=up:replay} 2024-02-10T06:57:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:55 smithi029 ceph-mon[53210]: pgmap v297: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 7.2 KiB/s wr, 3 op/s 2024-02-10T06:57:56.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:55 smithi029 ceph-mon[53210]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-10T06:57:56.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:55 smithi029 ceph-mon[53210]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T06:57:56.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:55 smithi029 ceph-mon[53210]: osdmap e140: 8 total, 8 up, 8 in 2024-02-10T06:57:56.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:55 smithi029 ceph-mon[53210]: Standby daemon mds.foofs.smithi008.vfwcte assigned to filesystem foofs as rank 0 2024-02-10T06:57:56.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:55 smithi029 ceph-mon[53210]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-10T06:57:56.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:55 smithi029 ceph-mon[53210]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-10T06:57:56.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:55 smithi029 ceph-mon[53210]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-02-10T06:57:56.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:55 smithi029 ceph-mon[53210]: fsmap foofs:1/1 {0=foofs.smithi008.vfwcte=up:replay} 2024-02-10T06:57:58.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:57 smithi008 ceph-mon[66415]: pgmap v299: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 204 B/s rd, 307 B/s wr, 0 op/s 2024-02-10T06:57:58.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:57 smithi008 ceph-mon[66415]: reconnect by client.14558 172.21.15.8:0/1464178136 after 0.00099999 2024-02-10T06:57:58.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:57 smithi008 ceph-mon[66415]: reconnect by client.24363 172.21.15.29:0/3804362279 after 0.00099999 2024-02-10T06:57:58.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:57 smithi008 ceph-mon[66415]: mds.? [v2:172.21.15.8:6834/2814645351,v1:172.21.15.8:6835/2814645351] up:reconnect 2024-02-10T06:57:58.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:57 smithi008 ceph-mon[66415]: fsmap foofs:1/1 {0=foofs.smithi008.vfwcte=up:reconnect} 2024-02-10T06:57:58.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:57 smithi029 ceph-mon[53210]: pgmap v299: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 204 B/s rd, 307 B/s wr, 0 op/s 2024-02-10T06:57:58.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:57 smithi029 ceph-mon[53210]: reconnect by client.14558 172.21.15.8:0/1464178136 after 0.00099999 2024-02-10T06:57:58.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:57 smithi029 ceph-mon[53210]: reconnect by client.24363 172.21.15.29:0/3804362279 after 0.00099999 2024-02-10T06:57:58.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:57 smithi029 ceph-mon[53210]: mds.? [v2:172.21.15.8:6834/2814645351,v1:172.21.15.8:6835/2814645351] up:reconnect 2024-02-10T06:57:58.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:57 smithi029 ceph-mon[53210]: fsmap foofs:1/1 {0=foofs.smithi008.vfwcte=up:reconnect} 2024-02-10T06:57:59.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:58 smithi029 ceph-mon[53210]: mds.? [v2:172.21.15.8:6834/2814645351,v1:172.21.15.8:6835/2814645351] up:rejoin 2024-02-10T06:57:59.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:58 smithi029 ceph-mon[53210]: fsmap foofs:1/1 {0=foofs.smithi008.vfwcte=up:rejoin} 2024-02-10T06:57:59.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:58 smithi029 ceph-mon[53210]: daemon mds.foofs.smithi008.vfwcte is now active in filesystem foofs as rank 0 2024-02-10T06:57:59.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:58 smithi008 ceph-mon[66415]: mds.? [v2:172.21.15.8:6834/2814645351,v1:172.21.15.8:6835/2814645351] up:rejoin 2024-02-10T06:57:59.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:58 smithi008 ceph-mon[66415]: fsmap foofs:1/1 {0=foofs.smithi008.vfwcte=up:rejoin} 2024-02-10T06:57:59.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:58 smithi008 ceph-mon[66415]: daemon mds.foofs.smithi008.vfwcte is now active in filesystem foofs as rank 0 2024-02-10T06:58:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:59 smithi029 ceph-mon[53210]: pgmap v300: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 307 B/s wr, 4 op/s 2024-02-10T06:58:00.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:59 smithi029 ceph-mon[53210]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-10T06:58:00.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:59 smithi029 ceph-mon[53210]: mds.? [v2:172.21.15.8:6834/2814645351,v1:172.21.15.8:6835/2814645351] up:active 2024-02-10T06:58:00.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:57:59 smithi029 ceph-mon[53210]: fsmap foofs:1 {0=foofs.smithi008.vfwcte=up:active} 2024-02-10T06:58:00.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:59 smithi008 ceph-mon[66415]: pgmap v300: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 307 B/s wr, 4 op/s 2024-02-10T06:58:00.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:59 smithi008 ceph-mon[66415]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-10T06:58:00.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:59 smithi008 ceph-mon[66415]: mds.? [v2:172.21.15.8:6834/2814645351,v1:172.21.15.8:6835/2814645351] up:active 2024-02-10T06:58:00.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:57:59 smithi008 ceph-mon[66415]: fsmap foofs:1 {0=foofs.smithi008.vfwcte=up:active} 2024-02-10T06:58:02.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:01 smithi008 ceph-mon[66415]: pgmap v301: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 409 B/s wr, 9 op/s 2024-02-10T06:58:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:01 smithi029 ceph-mon[53210]: pgmap v301: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 409 B/s wr, 9 op/s 2024-02-10T06:58:02.754 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:58:03.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:02 smithi008 ceph-mon[66415]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-10T06:58:03.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:02 smithi008 ceph-mon[66415]: Cluster is now healthy 2024-02-10T06:58:03.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:02 smithi008 ceph-mon[66415]: mds.? [v2:172.21.15.29:6832/182226001,v1:172.21.15.29:6833/182226001] up:boot 2024-02-10T06:58:03.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:02 smithi008 ceph-mon[66415]: fsmap foofs:1 {0=foofs.smithi008.vfwcte=up:active} 1 up:standby 2024-02-10T06:58:03.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:02 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi029.bwmcyt"}]: dispatch 2024-02-10T06:58:03.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:02 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:03.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:02 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:03.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:02 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:58:03.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:02 smithi029 ceph-mon[53210]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-10T06:58:03.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:02 smithi029 ceph-mon[53210]: Cluster is now healthy 2024-02-10T06:58:03.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:02 smithi029 ceph-mon[53210]: mds.? [v2:172.21.15.29:6832/182226001,v1:172.21.15.29:6833/182226001] up:boot 2024-02-10T06:58:03.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:02 smithi029 ceph-mon[53210]: fsmap foofs:1 {0=foofs.smithi008.vfwcte=up:active} 1 up:standby 2024-02-10T06:58:03.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:02 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi029.bwmcyt"}]: dispatch 2024-02-10T06:58:03.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:02 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:03.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:02 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:03.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:02 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:58:03.428 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:58:03.428 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (10m) 17s ago 20m 23.7M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:58:03.428 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (6m) 17s ago 20m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T06:58:03.428 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (6m) 41s ago 19m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T06:58:03.428 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (9m) 17s ago 19m 44.5M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:58:03.428 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (22s) 17s ago 13m 11.2M - 19.0.0-1361-g290212a1 c328ab9dcfc7 6c0c3a230f64 2024-02-10T06:58:03.428 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 starting - - - - 2024-02-10T06:58:03.428 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (9m) 17s ago 21m 510M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:58:03.428 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (8m) 41s ago 19m 452M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:58:03.428 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (7m) 17s ago 21m 60.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:58:03.428 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (7m) 41s ago 19m 45.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:58:03.429 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (13m) 17s ago 13m 254M - 4.0 e1d6a67b021e 4ce3bfdf5d2e 2024-02-10T06:58:03.429 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (13m) 41s ago 13m 84.7M - 4.0 e1d6a67b021e 2ee6202b6ada 2024-02-10T06:58:03.429 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (9m) 17s ago 19m 13.4M - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:58:03.429 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (9m) 41s ago 19m 17.6M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:58:03.429 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (5m) 17s ago 18m 378M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 acb0107eae6c 2024-02-10T06:58:03.429 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (4m) 17s ago 18m 337M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 09363f1c35ae 2024-02-10T06:58:03.429 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (4m) 17s ago 17m 337M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 bd6bbb022085 2024-02-10T06:58:03.429 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (3m) 17s ago 17m 384M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc3683d167e9 2024-02-10T06:58:03.429 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (2m) 41s ago 16m 344M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 2b1f7cbc27ce 2024-02-10T06:58:03.429 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (2m) 41s ago 16m 306M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 ea4052b4cafa 2024-02-10T06:58:03.429 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (82s) 41s ago 15m 258M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 d89cf7ba8295 2024-02-10T06:58:03.429 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (46s) 41s ago 15m 12.6M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 156ce7027d7a 2024-02-10T06:58:03.429 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (8m) 17s ago 18m 63.6M - 2.33.4 514e6a882f6e 193819b86dea 2024-02-10T06:58:03.837 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:58:03.837 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:58:03.837 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:58:03.837 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:58:03.838 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:58:03.838 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:58:03.838 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:58:03.838 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:58:03.838 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T06:58:03.838 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:58:03.838 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:58:03.838 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:58:03.838 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:58:03.838 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:58:03.838 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 14 2024-02-10T06:58:03.838 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:58:03.838 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:58:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:03 smithi029 ceph-mon[53210]: pgmap v302: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 409 B/s wr, 9 op/s 2024-02-10T06:58:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:03 smithi029 ceph-mon[53210]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:58:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:58:03.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:03 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/2811261901' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:04.179 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:58:04.179 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:58:04.180 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:58:04.180 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:58:04.180 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:58:04.180 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:58:04.180 INFO:teuthology.orchestra.run.smithi008.stdout: "osd", 2024-02-10T06:58:04.180 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-02-10T06:58:04.180 INFO:teuthology.orchestra.run.smithi008.stdout: "mon" 2024-02-10T06:58:04.180 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:58:04.180 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "15/23 daemons upgraded", 2024-02-10T06:58:04.180 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading mds daemons", 2024-02-10T06:58:04.180 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:58:04.180 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:58:04.181 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:03 smithi008 ceph-mon[66415]: pgmap v302: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 409 B/s wr, 9 op/s 2024-02-10T06:58:04.181 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:03 smithi008 ceph-mon[66415]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:58:04.181 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:58:04.181 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:03 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/2811261901' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:04.585 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:58:05.023 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:04 smithi029 ceph-mon[53210]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:58:05.023 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:04 smithi029 ceph-mon[53210]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:58:05.023 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:04 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/3305299954' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:58:05.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:04 smithi008 ceph-mon[66415]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:58:05.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:04 smithi008 ceph-mon[66415]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:58:05.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:04 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/3305299954' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:58:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:05 smithi029 ceph-mon[53210]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:58:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:05 smithi029 ceph-mon[53210]: pgmap v303: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 7.4 KiB/s wr, 14 op/s 2024-02-10T06:58:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:06.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:05 smithi008 ceph-mon[66415]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:58:06.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:05 smithi008 ceph-mon[66415]: pgmap v303: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 7.4 KiB/s wr, 14 op/s 2024-02-10T06:58:06.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:06.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:07 smithi029 ceph-mon[53210]: pgmap v304: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-02-10T06:58:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:07 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:07 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:08.028 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:07 smithi008 ceph-mon[66415]: pgmap v304: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-02-10T06:58:08.028 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:07 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:08.029 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:07 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:10.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:09 smithi008 ceph-mon[66415]: pgmap v305: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 7.3 KiB/s wr, 12 op/s 2024-02-10T06:58:10.135 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:09 smithi029 ceph-mon[53210]: pgmap v305: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 7.3 KiB/s wr, 12 op/s 2024-02-10T06:58:11.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: Detected new or changed devices on smithi029 2024-02-10T06:58:11.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:58:11.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:58:11.460 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.460 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.460 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:58:11.460 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.460 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.461 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.461 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.461 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.461 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: Upgrade: Setting container_image for all mds 2024-02-10T06:58:11.461 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.461 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi008.vfwcte"}]: dispatch 2024-02-10T06:58:11.462 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi008.vfwcte"}]': finished 2024-02-10T06:58:11.462 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi029.bwmcyt"}]: dispatch 2024-02-10T06:58:11.462 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi029.bwmcyt"}]': finished 2024-02-10T06:58:11.462 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.462 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: Upgrade: Setting container_image for all rgw 2024-02-10T06:58:11.462 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.463 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.463 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:58:11.463 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.463 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.463 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.464 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:58:11.464 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.464 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.464 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: Upgrade: Setting container_image for all iscsi 2024-02-10T06:58:11.464 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.464 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:11 smithi029 ceph-mon[53210]: pgmap v306: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 7.3 KiB/s wr, 9 op/s 2024-02-10T06:58:11.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: Detected new or changed devices on smithi029 2024-02-10T06:58:11.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:58:11.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:58:11.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:58:11.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.535 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: Upgrade: Setting container_image for all mds 2024-02-10T06:58:11.535 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.535 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi008.vfwcte"}]: dispatch 2024-02-10T06:58:11.535 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi008.vfwcte"}]': finished 2024-02-10T06:58:11.535 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi029.bwmcyt"}]: dispatch 2024-02-10T06:58:11.535 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi029.bwmcyt"}]': finished 2024-02-10T06:58:11.535 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.535 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: Upgrade: Setting container_image for all rgw 2024-02-10T06:58:11.535 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.535 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.535 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T06:58:11.535 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.536 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.536 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.536 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T06:58:11.536 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.536 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:11.536 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: Upgrade: Setting container_image for all iscsi 2024-02-10T06:58:11.536 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:11.536 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:11 smithi008 ceph-mon[66415]: pgmap v306: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 7.3 KiB/s wr, 9 op/s 2024-02-10T06:58:13.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:12 smithi008 ceph-mon[66415]: Upgrade: Updating nfs.foo.0.0.smithi008.pgujgu (1/2) 2024-02-10T06:58:13.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:12 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:13.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:12 smithi008 ceph-mon[66415]: Creating key for client.nfs.foo.0.0.smithi008.pgujgu 2024-02-10T06:58:13.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:12 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.pgujgu", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-10T06:58:13.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:12 smithi008 ceph-mon[66415]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-02-10T06:58:13.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:12 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-10T06:58:13.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:12 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-10T06:58:13.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:12 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:58:13.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:12 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-10T06:58:13.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:12 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-10T06:58:13.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:12 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.pgujgu-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:58:13.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:12 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:58:13.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:12 smithi029 ceph-mon[53210]: Upgrade: Updating nfs.foo.0.0.smithi008.pgujgu (1/2) 2024-02-10T06:58:13.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:12 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:13.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:12 smithi029 ceph-mon[53210]: Creating key for client.nfs.foo.0.0.smithi008.pgujgu 2024-02-10T06:58:13.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:12 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.pgujgu", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-10T06:58:13.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:12 smithi029 ceph-mon[53210]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-02-10T06:58:13.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:12 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-10T06:58:13.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:12 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-10T06:58:13.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:12 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:58:13.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:12 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-10T06:58:13.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:12 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-10T06:58:13.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:12 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.pgujgu-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:58:13.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:12 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:58:14.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:13 smithi008 ceph-mon[66415]: Rados config object exists: conf-nfs.foo 2024-02-10T06:58:14.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:13 smithi008 ceph-mon[66415]: Creating key for client.nfs.foo.0.0.smithi008.pgujgu-rgw 2024-02-10T06:58:14.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:13 smithi008 ceph-mon[66415]: Bind address in nfs.foo.0.0.smithi008.pgujgu's ganesha conf is defaulting to empty 2024-02-10T06:58:14.270 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:13 smithi008 ceph-mon[66415]: Deploying daemon nfs.foo.0.0.smithi008.pgujgu on smithi008 2024-02-10T06:58:14.270 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:13 smithi008 ceph-mon[66415]: pgmap v307: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.9 KiB/s rd, 7.2 KiB/s wr, 4 op/s 2024-02-10T06:58:14.282 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:13 smithi029 ceph-mon[53210]: Rados config object exists: conf-nfs.foo 2024-02-10T06:58:14.282 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:13 smithi029 ceph-mon[53210]: Creating key for client.nfs.foo.0.0.smithi008.pgujgu-rgw 2024-02-10T06:58:14.282 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:13 smithi029 ceph-mon[53210]: Bind address in nfs.foo.0.0.smithi008.pgujgu's ganesha conf is defaulting to empty 2024-02-10T06:58:14.282 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:13 smithi029 ceph-mon[53210]: Deploying daemon nfs.foo.0.0.smithi008.pgujgu on smithi008 2024-02-10T06:58:14.282 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:13 smithi029 ceph-mon[53210]: pgmap v307: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.9 KiB/s rd, 7.2 KiB/s wr, 4 op/s 2024-02-10T06:58:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:16 smithi029 ceph-mon[53210]: pgmap v308: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.9 KiB/s rd, 7.4 KiB/s wr, 5 op/s 2024-02-10T06:58:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:16 smithi029 ceph-mon[53210]: from='mds.? [v2:172.21.15.8:6834/2814645351,v1:172.21.15.8:6835/2814645351]' entity='mds.foofs.smithi008.vfwcte' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.8:0/1464178136"}]: dispatch 2024-02-10T06:58:16.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:16 smithi008 ceph-mon[66415]: pgmap v308: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.9 KiB/s rd, 7.4 KiB/s wr, 5 op/s 2024-02-10T06:58:16.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:16 smithi008 ceph-mon[66415]: from='mds.? [v2:172.21.15.8:6834/2814645351,v1:172.21.15.8:6835/2814645351]' entity='mds.foofs.smithi008.vfwcte' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.8:0/1464178136"}]: dispatch 2024-02-10T06:58:17.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:17 smithi008 ceph-mon[66415]: Evicting (and blocklisting) client session 14558 (172.21.15.8:0/1464178136) 2024-02-10T06:58:17.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:17 smithi008 ceph-mon[66415]: from='mds.? [v2:172.21.15.8:6834/2814645351,v1:172.21.15.8:6835/2814645351]' entity='mds.foofs.smithi008.vfwcte' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.8:0/1464178136"}]': finished 2024-02-10T06:58:17.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:17 smithi008 ceph-mon[66415]: osdmap e141: 8 total, 8 up, 8 in 2024-02-10T06:58:17.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:17.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:17 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:17.299 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:17 smithi029 ceph-mon[53210]: Evicting (and blocklisting) client session 14558 (172.21.15.8:0/1464178136) 2024-02-10T06:58:17.299 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:17 smithi029 ceph-mon[53210]: from='mds.? [v2:172.21.15.8:6834/2814645351,v1:172.21.15.8:6835/2814645351]' entity='mds.foofs.smithi008.vfwcte' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.8:0/1464178136"}]': finished 2024-02-10T06:58:17.300 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:17 smithi029 ceph-mon[53210]: osdmap e141: 8 total, 8 up, 8 in 2024-02-10T06:58:17.300 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:17.300 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:17 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:18.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:18 smithi029 ceph-mon[53210]: pgmap v310: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 204 B/s rd, 1.7 KiB/s wr, 0 op/s 2024-02-10T06:58:18.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:18.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi029.cvyqqw", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-10T06:58:18.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-10T06:58:18.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-10T06:58:18.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:58:18.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-10T06:58:18.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-10T06:58:18.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi029.cvyqqw-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:58:18.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:58:18.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:18 smithi008 ceph-mon[66415]: pgmap v310: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 204 B/s rd, 1.7 KiB/s wr, 0 op/s 2024-02-10T06:58:18.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:18.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi029.cvyqqw", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-10T06:58:18.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-10T06:58:18.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-10T06:58:18.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:58:18.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-10T06:58:18.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-10T06:58:18.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi029.cvyqqw-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T06:58:18.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:58:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:19 smithi029 ceph-mon[53210]: Upgrade: Updating nfs.foo.1.0.smithi029.cvyqqw (2/2) 2024-02-10T06:58:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:19 smithi029 ceph-mon[53210]: Creating key for client.nfs.foo.1.0.smithi029.cvyqqw 2024-02-10T06:58:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:19 smithi029 ceph-mon[53210]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-02-10T06:58:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:19 smithi029 ceph-mon[53210]: Rados config object exists: conf-nfs.foo 2024-02-10T06:58:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:19 smithi029 ceph-mon[53210]: Creating key for client.nfs.foo.1.0.smithi029.cvyqqw-rgw 2024-02-10T06:58:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:19 smithi029 ceph-mon[53210]: Bind address in nfs.foo.1.0.smithi029.cvyqqw's ganesha conf is defaulting to empty 2024-02-10T06:58:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:19 smithi029 ceph-mon[53210]: Deploying daemon nfs.foo.1.0.smithi029.cvyqqw on smithi029 2024-02-10T06:58:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:19 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:58:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:19 smithi008 ceph-mon[66415]: Upgrade: Updating nfs.foo.1.0.smithi029.cvyqqw (2/2) 2024-02-10T06:58:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:19 smithi008 ceph-mon[66415]: Creating key for client.nfs.foo.1.0.smithi029.cvyqqw 2024-02-10T06:58:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:19 smithi008 ceph-mon[66415]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-02-10T06:58:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:19 smithi008 ceph-mon[66415]: Rados config object exists: conf-nfs.foo 2024-02-10T06:58:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:19 smithi008 ceph-mon[66415]: Creating key for client.nfs.foo.1.0.smithi029.cvyqqw-rgw 2024-02-10T06:58:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:19 smithi008 ceph-mon[66415]: Bind address in nfs.foo.1.0.smithi029.cvyqqw's ganesha conf is defaulting to empty 2024-02-10T06:58:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:19 smithi008 ceph-mon[66415]: Deploying daemon nfs.foo.1.0.smithi029.cvyqqw on smithi029 2024-02-10T06:58:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:19 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:58:20.076 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:20 smithi029 ceph-mon[53210]: pgmap v311: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T06:58:20.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:20 smithi008 ceph-mon[66415]: pgmap v311: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T06:58:21.451 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:21 smithi029 ceph-mon[53210]: pgmap v312: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T06:58:21.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:21 smithi008 ceph-mon[66415]: pgmap v312: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T06:58:23.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:22 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:23.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:22 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:23.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:22 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:58:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:22 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:22 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:23.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:22 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:58:24.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:23 smithi008 ceph-mon[66415]: pgmap v313: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-02-10T06:58:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:23 smithi029 ceph-mon[53210]: pgmap v313: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-02-10T06:58:25.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:24 smithi008 ceph-mon[66415]: from='mds.? [v2:172.21.15.8:6834/2814645351,v1:172.21.15.8:6835/2814645351]' entity='mds.foofs.smithi008.vfwcte' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.29:0/3804362279"}]: dispatch 2024-02-10T06:58:25.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:24 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:25.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:24 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:24 smithi029 ceph-mon[53210]: from='mds.? [v2:172.21.15.8:6834/2814645351,v1:172.21.15.8:6835/2814645351]' entity='mds.foofs.smithi008.vfwcte' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.29:0/3804362279"}]: dispatch 2024-02-10T06:58:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:24 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:25.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:24 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:25 smithi029 ceph-mon[53210]: pgmap v314: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-02-10T06:58:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:25 smithi029 ceph-mon[53210]: Evicting (and blocklisting) client session 24363 (172.21.15.29:0/3804362279) 2024-02-10T06:58:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:25 smithi029 ceph-mon[53210]: from='mds.? [v2:172.21.15.8:6834/2814645351,v1:172.21.15.8:6835/2814645351]' entity='mds.foofs.smithi008.vfwcte' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.29:0/3804362279"}]': finished 2024-02-10T06:58:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:25 smithi029 ceph-mon[53210]: osdmap e142: 8 total, 8 up, 8 in 2024-02-10T06:58:26.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:25 smithi008 ceph-mon[66415]: pgmap v314: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-02-10T06:58:26.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:25 smithi008 ceph-mon[66415]: Evicting (and blocklisting) client session 24363 (172.21.15.29:0/3804362279) 2024-02-10T06:58:26.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:25 smithi008 ceph-mon[66415]: from='mds.? [v2:172.21.15.8:6834/2814645351,v1:172.21.15.8:6835/2814645351]' entity='mds.foofs.smithi008.vfwcte' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.29:0/3804362279"}]': finished 2024-02-10T06:58:26.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:25 smithi008 ceph-mon[66415]: osdmap e142: 8 total, 8 up, 8 in 2024-02-10T06:58:27.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:26 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:27.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:26 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:27.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:26 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:27.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:26 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:27 smithi029 ceph-mon[53210]: pgmap v316: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-02-10T06:58:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:28.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:27 smithi008 ceph-mon[66415]: pgmap v316: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-02-10T06:58:28.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:28.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:29.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:29 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:29.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:29 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:29.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:29 smithi029 ceph-mon[53210]: pgmap v317: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-02-10T06:58:29.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:29 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:29.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:29 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:29.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:29 smithi008 ceph-mon[66415]: pgmap v317: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-02-10T06:58:30.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:30 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:30.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:30 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:30.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:30 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:30.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:30 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:31.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:31 smithi008 ceph-mon[66415]: pgmap v318: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T06:58:31.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:31 smithi029 ceph-mon[53210]: pgmap v318: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T06:58:32.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:32.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:32.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:58:32.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:58:32.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:32.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:32.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:58:32.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:32.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:32.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:32.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:32.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:32.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:32.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:32.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:32.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:32.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:32.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:32.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: Upgrade: Setting container_image for all nfs 2024-02-10T06:58:32.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:32.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi008.pgujgu"}]: dispatch 2024-02-10T06:58:32.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi008.pgujgu"}]': finished 2024-02-10T06:58:32.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi029.cvyqqw"}]: dispatch 2024-02-10T06:58:32.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi029.cvyqqw"}]': finished 2024-02-10T06:58:32.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:32.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:58:32.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:32 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:58:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:58:33.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:33.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:33.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:58:33.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:33.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:33.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:33.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:33.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:33.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:33.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:33.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:33.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:33.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:33.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:33.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: Upgrade: Setting container_image for all nfs 2024-02-10T06:58:33.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:33.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi008.pgujgu"}]: dispatch 2024-02-10T06:58:33.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi008.pgujgu"}]': finished 2024-02-10T06:58:33.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi029.cvyqqw"}]: dispatch 2024-02-10T06:58:33.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi029.cvyqqw"}]': finished 2024-02-10T06:58:33.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:33.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: Upgrade: Setting container_image for all nvmeof 2024-02-10T06:58:33.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:32 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:34.206 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:33 smithi008 ceph-mon[66415]: pgmap v319: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 204 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:58:34.206 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:33 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:58:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:33 smithi029 ceph-mon[53210]: pgmap v319: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 204 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:58:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:33 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:58:34.954 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:58:35.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:34 smithi008 ceph-mon[66415]: Upgrade: Updating node-exporter.smithi008 (1/2) 2024-02-10T06:58:35.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:34 smithi008 ceph-mon[66415]: Deploying daemon node-exporter.smithi008 on smithi008 2024-02-10T06:58:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:34 smithi029 ceph-mon[53210]: Upgrade: Updating node-exporter.smithi008 (1/2) 2024-02-10T06:58:35.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:34 smithi029 ceph-mon[53210]: Deploying daemon node-exporter.smithi008 on smithi008 2024-02-10T06:58:35.670 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:58:35.670 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (10m) 8s ago 20m 23.4M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:58:35.670 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (6m) 8s ago 20m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T06:58:35.670 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (6m) 10s ago 19m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T06:58:35.670 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (10m) 8s ago 20m 50.5M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:58:35.670 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (55s) 8s ago 14m 33.8M - 19.0.0-1361-g290212a1 c328ab9dcfc7 6c0c3a230f64 2024-02-10T06:58:35.670 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (34s) 10s ago 14m 14.6M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7c69029ef9bb 2024-02-10T06:58:35.670 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (9m) 8s ago 22m 511M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:58:35.670 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (8m) 10s ago 19m 453M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:58:35.670 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (8m) 8s ago 22m 62.7M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:58:35.670 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (7m) 10s ago 19m 47.7M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:58:35.671 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (20s) 8s ago 14m 71.6M - 5.7 c328ab9dcfc7 00f651c4baff 2024-02-10T06:58:35.671 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (14s) 10s ago 14m 21.6M - 5.7 c328ab9dcfc7 f754175aa820 2024-02-10T06:58:35.671 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (10m) 8s ago 20m 17.3M - 1.3.1 1dbe0e931976 073f3ca2f472 2024-02-10T06:58:35.671 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (9m) 10s ago 19m 17.6M - 1.3.1 1dbe0e931976 a05a81ee099b 2024-02-10T06:58:35.671 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (6m) 8s ago 19m 379M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 acb0107eae6c 2024-02-10T06:58:35.671 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (5m) 8s ago 18m 340M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 09363f1c35ae 2024-02-10T06:58:35.671 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (4m) 8s ago 18m 337M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 bd6bbb022085 2024-02-10T06:58:35.671 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (3m) 8s ago 17m 386M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc3683d167e9 2024-02-10T06:58:35.671 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (3m) 10s ago 17m 346M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 2b1f7cbc27ce 2024-02-10T06:58:35.671 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (2m) 10s ago 16m 310M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 ea4052b4cafa 2024-02-10T06:58:35.671 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (115s) 10s ago 16m 246M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 d89cf7ba8295 2024-02-10T06:58:35.671 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (78s) 10s ago 16m 283M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 156ce7027d7a 2024-02-10T06:58:35.671 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (8m) 8s ago 19m 63.6M - 2.33.4 514e6a882f6e 193819b86dea 2024-02-10T06:58:36.076 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:58:36.077 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:58:36.077 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:58:36.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:58:36.077 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:58:36.077 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:58:36.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:58:36.077 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:58:36.077 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T06:58:36.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:58:36.077 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:58:36.077 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:58:36.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:58:36.077 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:58:36.078 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 14 2024-02-10T06:58:36.078 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:58:36.078 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:58:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:35 smithi029 ceph-mon[53210]: pgmap v320: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 204 B/s rd, 409 B/s wr, 0 op/s 2024-02-10T06:58:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:35 smithi029 ceph-mon[53210]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:58:36.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:35 smithi008 ceph-mon[66415]: pgmap v320: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 204 B/s rd, 409 B/s wr, 0 op/s 2024-02-10T06:58:36.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:35 smithi008 ceph-mon[66415]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:58:36.418 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:58:36.419 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:58:36.419 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:58:36.419 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:58:36.419 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:58:36.419 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-02-10T06:58:36.419 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-02-10T06:58:36.419 INFO:teuthology.orchestra.run.smithi008.stdout: "nfs", 2024-02-10T06:58:36.419 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:58:36.419 INFO:teuthology.orchestra.run.smithi008.stdout: "osd", 2024-02-10T06:58:36.419 INFO:teuthology.orchestra.run.smithi008.stdout: "mds" 2024-02-10T06:58:36.419 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:58:36.419 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "18/23 daemons upgraded", 2024-02-10T06:58:36.420 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading node-exporter daemons", 2024-02-10T06:58:36.420 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:58:36.420 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:58:36.850 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:58:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:36 smithi029 ceph-mon[53210]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:58:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:36 smithi029 ceph-mon[53210]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:58:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:36 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/4001263047' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:37.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:36 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/4282854769' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:58:37.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:36 smithi008 ceph-mon[66415]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:58:37.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:36 smithi008 ceph-mon[66415]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:58:37.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:36 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/4001263047' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:37.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:36 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/4282854769' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:58:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:37 smithi029 ceph-mon[53210]: pgmap v321: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 89 B/s rd, 357 B/s wr, 0 op/s 2024-02-10T06:58:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:37 smithi029 ceph-mon[53210]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:58:38.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:37 smithi008 ceph-mon[66415]: pgmap v321: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 89 B/s rd, 357 B/s wr, 0 op/s 2024-02-10T06:58:38.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:37 smithi008 ceph-mon[66415]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:58:39.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:39 smithi029 ceph-mon[53210]: pgmap v322: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 341 B/s wr, 0 op/s 2024-02-10T06:58:39.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:39 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:39.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:39 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:40.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:39 smithi008 ceph-mon[66415]: pgmap v322: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 341 B/s wr, 0 op/s 2024-02-10T06:58:40.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:39 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:40.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:39 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:41.873 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:41 smithi029 ceph-mon[53210]: pgmap v323: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:58:41.873 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:41 smithi029 ceph-mon[53210]: Upgrade: Updating node-exporter.smithi029 (2/2) 2024-02-10T06:58:41.873 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:41 smithi029 ceph-mon[53210]: Deploying daemon node-exporter.smithi029 on smithi029 2024-02-10T06:58:42.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:41 smithi008 ceph-mon[66415]: pgmap v323: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:58:42.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:41 smithi008 ceph-mon[66415]: Upgrade: Updating node-exporter.smithi029 (2/2) 2024-02-10T06:58:42.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:41 smithi008 ceph-mon[66415]: Deploying daemon node-exporter.smithi029 on smithi029 2024-02-10T06:58:43.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:43 smithi029 ceph-mon[53210]: pgmap v324: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:58:44.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:43 smithi008 ceph-mon[66415]: pgmap v324: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:58:46.099 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:45 smithi029 ceph-mon[53210]: pgmap v325: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:58:46.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:45 smithi008 ceph-mon[66415]: pgmap v325: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:58:47.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:47 smithi008 ceph-mon[66415]: pgmap v326: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:58:47.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:47.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:47.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:58:47.786 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:47 smithi029 ceph-mon[53210]: pgmap v326: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:58:47.786 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:47.786 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:47.786 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:58:48.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:58:48.998 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:58:50.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:49 smithi008 ceph-mon[66415]: pgmap v327: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:58:50.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:49 smithi029 ceph-mon[53210]: pgmap v327: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:58:51.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:50 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:51.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:50 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:51.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:50 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:51.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:50 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:51 smithi029 ceph-mon[53210]: pgmap v328: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:58:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:51 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:52.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:51 smithi008 ceph-mon[66415]: pgmap v328: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:58:52.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:52.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:52.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:52.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:51 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:54.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:53 smithi008 ceph-mon[66415]: pgmap v329: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:58:54.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:53 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:54.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:53 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:54.089 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:53 smithi029 ceph-mon[53210]: pgmap v329: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:58:54.089 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:53 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:54.089 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:53 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:55.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:55 smithi008 ceph-mon[66415]: pgmap v330: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:58:55.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:55 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:55.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:55 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:55.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:55 smithi029 ceph-mon[53210]: pgmap v330: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:58:55.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:55 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:55.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:55 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:57.449 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: pgmap v331: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:58:57.449 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:57.449 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:57.449 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:58:57.449 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:58:57.449 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:57.449 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:57.449 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:58:57.449 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.449 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.449 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.449 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:57 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.656 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: pgmap v331: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:58:57.656 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:57.656 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:57.656 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:58:57.656 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:58:57.657 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:57.657 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:58:57.657 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:58:57.657 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.657 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.657 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.657 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.657 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.657 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.657 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.657 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.657 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.657 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.657 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.657 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:58:57.658 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:57 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:00.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:59 smithi008 ceph-mon[66415]: Upgrade: Updating prometheus.smithi008 2024-02-10T06:59:00.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:59 smithi008 ceph-mon[66415]: pgmap v332: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:59:00.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:58:59 smithi008 ceph-mon[66415]: Deploying daemon prometheus.smithi008 on smithi008 2024-02-10T06:59:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:59 smithi029 ceph-mon[53210]: Upgrade: Updating prometheus.smithi008 2024-02-10T06:59:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:59 smithi029 ceph-mon[53210]: pgmap v332: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:59:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:58:59 smithi029 ceph-mon[53210]: Deploying daemon prometheus.smithi008 on smithi008 2024-02-10T06:59:01.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:01 smithi008 ceph-mon[66415]: pgmap v333: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:01 smithi029 ceph-mon[53210]: pgmap v333: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:03.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:03 smithi008 ceph-mon[66415]: pgmap v334: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:03.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:59:03.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:03.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:03.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:59:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:03 smithi029 ceph-mon[53210]: pgmap v334: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:59:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:59:05.274 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:05.274 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:05.274 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:05.274 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:05 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:05.274 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:05 smithi008 ceph-mon[66415]: pgmap v335: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:05.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:05.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:05.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:05.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:05 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:05.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:05 smithi029 ceph-mon[53210]: pgmap v335: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:07.217 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:59:07.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:07 smithi008 ceph-mon[66415]: pgmap v336: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:07.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:07 smithi029 ceph-mon[53210]: pgmap v336: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:07.905 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:59:07.905 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (11m) 16s ago 21m 23.4M - 0.23.0 ba2b418f427c 38ee29c84ce4 2024-02-10T06:59:07.905 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (7m) 16s ago 21m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T06:59:07.905 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (7m) 18s ago 20m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T06:59:07.906 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (11m) 16s ago 20m 50.9M - 8.3.5 dad864ee21e9 ec85a4cd1264 2024-02-10T06:59:07.906 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (87s) 16s ago 15m 34.0M - 19.0.0-1361-g290212a1 c328ab9dcfc7 6c0c3a230f64 2024-02-10T06:59:07.906 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (66s) 18s ago 14m 14.8M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7c69029ef9bb 2024-02-10T06:59:07.906 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (10m) 16s ago 22m 512M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:59:07.906 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (9m) 18s ago 20m 453M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:59:07.906 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (9m) 16s ago 22m 62.9M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:59:07.906 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (8m) 18s ago 20m 46.7M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:59:07.906 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (52s) 16s ago 14m 71.7M - 5.7 c328ab9dcfc7 00f651c4baff 2024-02-10T06:59:07.906 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (46s) 18s ago 14m 71.5M - 5.7 c328ab9dcfc7 f754175aa820 2024-02-10T06:59:07.906 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (29s) 16s ago 20m 8070k - 1.5.0 0da6a335fe13 6fdbf8ce9613 2024-02-10T06:59:07.906 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (21s) 18s ago 20m 5440k - 1.5.0 0da6a335fe13 70d1e39d65c1 2024-02-10T06:59:07.906 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (6m) 16s ago 19m 379M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 acb0107eae6c 2024-02-10T06:59:07.906 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (5m) 16s ago 19m 339M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 09363f1c35ae 2024-02-10T06:59:07.906 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (5m) 16s ago 18m 338M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 bd6bbb022085 2024-02-10T06:59:07.907 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (4m) 16s ago 18m 387M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc3683d167e9 2024-02-10T06:59:07.907 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (3m) 18s ago 17m 346M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 2b1f7cbc27ce 2024-02-10T06:59:07.907 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (3m) 18s ago 17m 310M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 ea4052b4cafa 2024-02-10T06:59:07.907 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (2m) 18s ago 17m 246M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 d89cf7ba8295 2024-02-10T06:59:07.907 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (110s) 18s ago 16m 284M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 156ce7027d7a 2024-02-10T06:59:07.907 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 starting - - - - 2024-02-10T06:59:08.323 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:59:08.323 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:59:08.323 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:59:08.323 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:59:08.324 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:59:08.324 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:59:08.324 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:59:08.324 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:59:08.324 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T06:59:08.324 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:59:08.324 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:59:08.324 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:59:08.324 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:59:08.324 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:59:08.324 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 14 2024-02-10T06:59:08.324 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:59:08.324 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:59:08.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:08 smithi008 ceph-mon[66415]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:59:08.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:08 smithi008 ceph-mon[66415]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:59:08.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:08 smithi008 ceph-mon[66415]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:59:08.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:08 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/3097960573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:08.697 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:59:08.697 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:59:08.697 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:59:08.697 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:59:08.697 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:59:08.697 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-02-10T06:59:08.698 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-02-10T06:59:08.698 INFO:teuthology.orchestra.run.smithi008.stdout: "nfs", 2024-02-10T06:59:08.698 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:59:08.698 INFO:teuthology.orchestra.run.smithi008.stdout: "osd", 2024-02-10T06:59:08.698 INFO:teuthology.orchestra.run.smithi008.stdout: "mds" 2024-02-10T06:59:08.698 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:59:08.698 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "18/23 daemons upgraded", 2024-02-10T06:59:08.698 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading prometheus daemons", 2024-02-10T06:59:08.698 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:59:08.698 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:59:08.707 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:08 smithi029 ceph-mon[53210]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:59:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:08 smithi029 ceph-mon[53210]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:59:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:08 smithi029 ceph-mon[53210]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:59:08.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:08 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/3097960573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:09.112 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:59:10.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:09 smithi008 ceph-mon[66415]: pgmap v337: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:59:10.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:09 smithi008 ceph-mon[66415]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:59:10.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:09 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/3025324030' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:59:10.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:09 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:10.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:09 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:10.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:09 smithi029 ceph-mon[53210]: pgmap v337: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:59:10.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:09 smithi029 ceph-mon[53210]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:59:10.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:09 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/3025324030' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:59:10.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:09 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:10.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:09 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:11 smithi008 ceph-mon[66415]: pgmap v338: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:11.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:11 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:11.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:11 smithi029 ceph-mon[53210]: pgmap v338: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:11.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:11.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:11 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:13.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:13 smithi008 ceph-mon[66415]: pgmap v339: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:13.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:13 smithi029 ceph-mon[53210]: pgmap v339: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:15.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:15.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:15.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:59:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:59:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:59:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:59:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:14 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:15.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:15.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:59:15.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:59:15.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:15.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:15.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:59:15.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:59:15.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:15.459 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:14 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:16.249 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:15 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:59:16.249 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:15 smithi008 ceph-mon[66415]: pgmap v340: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:16.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:15 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:59:16.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:15 smithi029 ceph-mon[53210]: pgmap v340: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:17.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:16 smithi008 ceph-mon[66415]: Upgrade: Updating alertmanager.smithi008 2024-02-10T06:59:17.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:16 smithi008 ceph-mon[66415]: Deploying daemon alertmanager.smithi008 on smithi008 2024-02-10T06:59:17.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:16 smithi029 ceph-mon[53210]: Upgrade: Updating alertmanager.smithi008 2024-02-10T06:59:17.359 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:16 smithi029 ceph-mon[53210]: Deploying daemon alertmanager.smithi008 on smithi008 2024-02-10T06:59:17.998 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:17 smithi008 ceph-mon[66415]: pgmap v341: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:18.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:17 smithi029 ceph-mon[53210]: pgmap v341: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:59:19.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:59:20.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:19 smithi008 ceph-mon[66415]: pgmap v342: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:59:20.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:19 smithi029 ceph-mon[53210]: pgmap v342: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:59:22.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:21 smithi008 ceph-mon[66415]: pgmap v343: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:22.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:21 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:22.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:21 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:22.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:21 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:59:22.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:21 smithi029 ceph-mon[53210]: pgmap v343: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:22.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:21 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:22.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:21 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:22.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:21 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:59:24.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:23 smithi008 ceph-mon[66415]: pgmap v344: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:24.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:23 smithi029 ceph-mon[53210]: pgmap v344: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:26.080 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:25 smithi008 ceph-mon[66415]: pgmap v345: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:26.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:25 smithi029 ceph-mon[53210]: pgmap v345: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:27.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:27.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:27 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:27.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:27 smithi029 ceph-mon[53210]: pgmap v346: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:27.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:27.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:27 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:27.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:27 smithi008 ceph-mon[66415]: pgmap v346: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:28.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:28 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:28.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:28 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:28.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:28 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:28.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:28 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:29.997 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:29 smithi008 ceph-mon[66415]: pgmap v347: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:59:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:29 smithi029 ceph-mon[53210]: pgmap v347: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:59:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: pgmap v348: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:59:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:59:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:59:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:59:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:59:31.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:31 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: pgmap v348: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:31.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:31.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:31.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:59:31.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:59:31.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:31.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:31.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:59:31.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:59:31.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:59:31.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.959 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:31.960 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:31 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:33 smithi029 ceph-mon[53210]: Upgrade: Updating grafana.smithi008 2024-02-10T06:59:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:33 smithi029 ceph-mon[53210]: Deploying daemon grafana.smithi008 on smithi008 2024-02-10T06:59:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:33 smithi029 ceph-mon[53210]: pgmap v349: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:33.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:33 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:59:34.003 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:33 smithi008 ceph-mon[66415]: Upgrade: Updating grafana.smithi008 2024-02-10T06:59:34.003 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:33 smithi008 ceph-mon[66415]: Deploying daemon grafana.smithi008 on smithi008 2024-02-10T06:59:34.003 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:33 smithi008 ceph-mon[66415]: pgmap v349: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:34.003 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:33 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:59:35.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:35 smithi008 ceph-mon[66415]: pgmap v350: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:35 smithi029 ceph-mon[53210]: pgmap v350: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:38.029 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:37 smithi008 ceph-mon[66415]: pgmap v351: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:38.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:37 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:38.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:37 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:38.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:37 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:59:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:37 smithi029 ceph-mon[53210]: pgmap v351: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:37 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:37 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:38.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:37 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:59:39.504 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T06:59:40.021 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:39 smithi008 ceph-mon[66415]: pgmap v352: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:59:40.199 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:59:40.199 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (19s) 13s ago 21m 17.2M - 0.25.0 c8568f914cd2 80540098fb8f 2024-02-10T06:59:40.199 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (8m) 13s ago 21m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T06:59:40.199 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (7m) 50s ago 21m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T06:59:40.199 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 starting - - - - 2024-02-10T06:59:40.199 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (119s) 13s ago 15m 34.3M - 19.0.0-1361-g290212a1 c328ab9dcfc7 6c0c3a230f64 2024-02-10T06:59:40.199 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (98s) 50s ago 15m 14.8M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7c69029ef9bb 2024-02-10T06:59:40.199 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (10m) 13s ago 23m 514M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T06:59:40.199 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (9m) 50s ago 20m 453M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T06:59:40.199 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (9m) 13s ago 23m 64.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T06:59:40.200 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (9m) 50s ago 20m 46.7M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T06:59:40.200 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (84s) 13s ago 15m 72.6M - 5.7 c328ab9dcfc7 00f651c4baff 2024-02-10T06:59:40.200 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (78s) 50s ago 15m 71.5M - 5.7 c328ab9dcfc7 f754175aa820 2024-02-10T06:59:40.200 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (61s) 13s ago 21m 8506k - 1.5.0 0da6a335fe13 6fdbf8ce9613 2024-02-10T06:59:40.200 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (54s) 50s ago 20m 5440k - 1.5.0 0da6a335fe13 70d1e39d65c1 2024-02-10T06:59:40.200 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (7m) 13s ago 20m 380M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 acb0107eae6c 2024-02-10T06:59:40.200 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (6m) 13s ago 19m 339M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 09363f1c35ae 2024-02-10T06:59:40.200 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (5m) 13s ago 19m 339M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 bd6bbb022085 2024-02-10T06:59:40.200 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (4m) 13s ago 18m 388M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc3683d167e9 2024-02-10T06:59:40.200 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (4m) 50s ago 18m 346M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 2b1f7cbc27ce 2024-02-10T06:59:40.200 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (3m) 50s ago 18m 310M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 ea4052b4cafa 2024-02-10T06:59:40.201 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (2m) 50s ago 17m 246M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 d89cf7ba8295 2024-02-10T06:59:40.201 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (2m) 50s ago 17m 284M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 156ce7027d7a 2024-02-10T06:59:40.201 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (37s) 13s ago 20m 46.4M - 2.43.0 a07b618ecd1d faaf1777c75f 2024-02-10T06:59:40.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:39 smithi029 ceph-mon[53210]: pgmap v352: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:59:40.622 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:59:40.622 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T06:59:40.622 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:59:40.622 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:59:40.622 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T06:59:40.622 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:59:40.623 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:59:40.623 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T06:59:40.623 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T06:59:40.623 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:59:40.623 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T06:59:40.623 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T06:59:40.623 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T06:59:40.623 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T06:59:40.623 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 14 2024-02-10T06:59:40.623 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T06:59:40.623 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:59:40.929 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:40 smithi008 ceph-mon[66415]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:59:40.929 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:40 smithi008 ceph-mon[66415]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:59:40.929 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:40 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/3460008656' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:40.968 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T06:59:40.968 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:59:40.968 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-02-10T06:59:40.968 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:59:40.968 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-02-10T06:59:40.968 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-02-10T06:59:40.968 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-02-10T06:59:40.968 INFO:teuthology.orchestra.run.smithi008.stdout: "nfs", 2024-02-10T06:59:40.968 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr", 2024-02-10T06:59:40.968 INFO:teuthology.orchestra.run.smithi008.stdout: "osd", 2024-02-10T06:59:40.968 INFO:teuthology.orchestra.run.smithi008.stdout: "mds" 2024-02-10T06:59:40.969 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T06:59:40.969 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "18/23 daemons upgraded", 2024-02-10T06:59:40.969 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading grafana daemons", 2024-02-10T06:59:40.969 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T06:59:40.969 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T06:59:41.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:40 smithi029 ceph-mon[53210]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:59:41.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:40 smithi029 ceph-mon[53210]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:59:41.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:40 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/3460008656' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:41.433 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T06:59:42.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:41 smithi008 ceph-mon[66415]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:59:42.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:41 smithi008 ceph-mon[66415]: pgmap v353: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:42.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:41 smithi008 ceph-mon[66415]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:59:42.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:41 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/3058521639' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:59:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:41 smithi029 ceph-mon[53210]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:59:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:41 smithi029 ceph-mon[53210]: pgmap v353: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:41 smithi029 ceph-mon[53210]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:59:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:41 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/3058521639' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:59:43.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:43 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:43.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:43 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:43.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:43 smithi029 ceph-mon[53210]: pgmap v354: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:43.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:43 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:43.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:43 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:43.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:43 smithi008 ceph-mon[66415]: pgmap v354: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:44.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:44 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:44.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:44 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:44.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:44 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:44.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:44 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:45.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:45 smithi008 ceph-mon[66415]: pgmap v355: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:45.708 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:45 smithi029 ceph-mon[53210]: pgmap v355: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:48.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: pgmap v356: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:48.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:59:48.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:59:48.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:59:48.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:59:48.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.210 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:59:48.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:59:48.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:59:48.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:59:48.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:59:48.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:59:48.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:59:48.211 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-10T06:59:48.212 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:59:48.212 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:59:48.212 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:59:48.212 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-10T06:59:48.212 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:59:48.212 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:59:48.212 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:59:48.212 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:59:48.212 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:59:48.212 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:59:48.212 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T06:59:48.212 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:59:48.212 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:59:48.212 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:59:48.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:59:48.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:59:48.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:59:48.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:59:48.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:59:48.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:59:48.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:59:48.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:59:48.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:59:48.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:59:48.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:59:48.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:59:48.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.213 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:59:48.214 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:59:48.214 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:59:48.214 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.214 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: pgmap v356: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:48.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:59:48.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:59:48.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:59:48.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:59:48.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:59:48.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T06:59:48.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T06:59:48.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:59:48.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T06:59:48.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T06:59:48.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T06:59:48.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T06:59:48.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-10T06:59:48.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T06:59:48.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T06:59:48.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T06:59:48.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-10T06:59:48.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T06:59:48.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T06:59:48.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:59:48.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T06:59:48.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T06:59:48.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T06:59:48.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T06:59:48.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T06:59:48.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T06:59:48.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T06:59:48.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T06:59:48.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:59:48.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:59:48.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:59:48.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:59:48.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:59:48.288 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T06:59:48.288 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T06:59:48.288 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T06:59:48.288 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:59:48.288 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:59:48.288 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:59:48.288 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.288 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.288 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:59:48.288 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:59:48.288 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:59:48.288 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:48.288 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:49.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:48 smithi029 ceph-mon[53210]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:59:49.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:48 smithi029 ceph-mon[53210]: Upgrade: Finalizing container_image settings 2024-02-10T06:59:49.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:48 smithi029 ceph-mon[53210]: Upgrade: Complete! 2024-02-10T06:59:49.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:59:49.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:49.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:48 smithi008 ceph-mon[66415]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:59:49.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:48 smithi008 ceph-mon[66415]: Upgrade: Finalizing container_image settings 2024-02-10T06:59:49.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:48 smithi008 ceph-mon[66415]: Upgrade: Complete! 2024-02-10T06:59:49.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:59:49.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T06:59:50.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:49 smithi029 ceph-mon[53210]: pgmap v357: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:59:50.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:49 smithi008 ceph-mon[66415]: pgmap v357: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T06:59:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:51 smithi029 ceph-mon[53210]: pgmap v358: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:52.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:51 smithi008 ceph-mon[66415]: pgmap v358: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:53 smithi029 ceph-mon[53210]: pgmap v359: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:54.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:53 smithi008 ceph-mon[66415]: pgmap v359: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:55 smithi029 ceph-mon[53210]: pgmap v360: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:56.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:55 smithi008 ceph-mon[66415]: pgmap v360: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:58.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:57 smithi008 ceph-mon[66415]: pgmap v361: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:59:58.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:57 smithi029 ceph-mon[53210]: pgmap v361: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:00.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 06:59:59 smithi008 ceph-mon[66415]: pgmap v362: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:00:00.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 06:59:59 smithi029 ceph-mon[53210]: pgmap v362: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:00:01.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:00 smithi008 ceph-mon[66415]: overall HEALTH_OK 2024-02-10T07:00:01.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:00 smithi029 ceph-mon[53210]: overall HEALTH_OK 2024-02-10T07:00:02.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:01 smithi008 ceph-mon[66415]: pgmap v363: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:02.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:01 smithi029 ceph-mon[53210]: pgmap v363: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:04.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:03 smithi008 ceph-mon[66415]: pgmap v364: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:04.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:00:04.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:03 smithi029 ceph-mon[53210]: pgmap v364: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:04.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:00:06.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:05 smithi008 ceph-mon[66415]: pgmap v365: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:06.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:05 smithi029 ceph-mon[53210]: pgmap v365: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:08.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:07 smithi008 ceph-mon[66415]: pgmap v366: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:08.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:07 smithi029 ceph-mon[53210]: pgmap v366: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:10.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:09 smithi008 ceph-mon[66415]: pgmap v367: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:00:10.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:09 smithi029 ceph-mon[53210]: pgmap v367: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:00:12.121 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:11 smithi008 ceph-mon[66415]: pgmap v368: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:12.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:11 smithi029 ceph-mon[53210]: pgmap v368: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:12.501 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T07:00:13.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:12 smithi008 ceph-mon[66415]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:00:13.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:12 smithi029 ceph-mon[53210]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:00:14.177 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:13 smithi008 ceph-mon[66415]: pgmap v369: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:14.428 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:00:14.429 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (53s) 32s ago 22m 17.2M - 0.25.0 c8568f914cd2 80540098fb8f 2024-02-10T07:00:14.429 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (8m) 32s ago 22m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T07:00:14.429 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (8m) 84s ago 21m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T07:00:14.429 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (37s) 32s ago 22m 56.7M - 9.4.12 a0ea5e3c46d9 9675a3298934 2024-02-10T07:00:14.429 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (2m) 32s ago 16m 34.4M - 19.0.0-1361-g290212a1 c328ab9dcfc7 6c0c3a230f64 2024-02-10T07:00:14.429 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (2m) 84s ago 16m 14.8M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7c69029ef9bb 2024-02-10T07:00:14.429 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (11m) 32s ago 23m 514M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T07:00:14.429 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (10m) 84s ago 21m 453M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T07:00:14.429 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (10m) 32s ago 23m 64.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T07:00:14.430 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (9m) 84s ago 21m 46.7M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T07:00:14.430 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (118s) 32s ago 15m 72.6M - 5.7 c328ab9dcfc7 00f651c4baff 2024-02-10T07:00:14.430 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (113s) 84s ago 15m 71.5M - 5.7 c328ab9dcfc7 f754175aa820 2024-02-10T07:00:14.430 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (96s) 32s ago 21m 8875k - 1.5.0 0da6a335fe13 6fdbf8ce9613 2024-02-10T07:00:14.430 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (88s) 84s ago 21m 5440k - 1.5.0 0da6a335fe13 70d1e39d65c1 2024-02-10T07:00:14.430 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (7m) 32s ago 20m 380M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 acb0107eae6c 2024-02-10T07:00:14.430 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (7m) 32s ago 20m 340M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 09363f1c35ae 2024-02-10T07:00:14.430 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (6m) 32s ago 19m 339M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 bd6bbb022085 2024-02-10T07:00:14.430 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (5m) 32s ago 19m 388M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc3683d167e9 2024-02-10T07:00:14.430 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (4m) 84s ago 18m 346M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 2b1f7cbc27ce 2024-02-10T07:00:14.430 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (4m) 84s ago 18m 310M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 ea4052b4cafa 2024-02-10T07:00:14.431 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (3m) 84s ago 18m 246M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 d89cf7ba8295 2024-02-10T07:00:14.431 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (2m) 84s ago 17m 284M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 156ce7027d7a 2024-02-10T07:00:14.431 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (71s) 32s ago 21m 47.6M - 2.43.0 a07b618ecd1d faaf1777c75f 2024-02-10T07:00:14.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:13 smithi029 ceph-mon[53210]: pgmap v369: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:15.160 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions' 2024-02-10T07:00:16.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:15 smithi008 ceph-mon[66415]: pgmap v370: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:16.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:15 smithi008 ceph-mon[66415]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:00:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:15 smithi029 ceph-mon[53210]: pgmap v370: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:15 smithi029 ceph-mon[53210]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:00:17.211 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:00:17.211 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T07:00:17.211 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:00:17.211 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:00:17.211 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T07:00:17.211 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:00:17.212 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:00:17.212 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T07:00:17.212 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T07:00:17.212 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:00:17.212 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T07:00:17.212 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:00:17.212 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:00:17.212 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T07:00:17.212 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 14 2024-02-10T07:00:17.212 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:00:17.212 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:00:17.856 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-02-10T07:00:18.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:17 smithi008 ceph-mon[66415]: pgmap v371: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:18.172 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:17 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/2647652949' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:00:18.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:17 smithi029 ceph-mon[53210]: pgmap v371: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:18.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:17 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/2647652949' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:00:19.172 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:00:19.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:00:19.484 INFO:teuthology.orchestra.run.smithi008.stdout:wait for servicemap items w/ changing names to refresh 2024-02-10T07:00:20.091 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:19 smithi008 ceph-mon[66415]: pgmap v372: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:00:20.115 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'sleep 60' 2024-02-10T07:00:20.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:19 smithi029 ceph-mon[53210]: pgmap v372: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:00:22.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:22 smithi008 ceph-mon[66415]: pgmap v373: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:22.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:22 smithi029 ceph-mon[53210]: pgmap v373: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:24.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:24 smithi008 ceph-mon[66415]: pgmap v374: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:24.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:24 smithi029 ceph-mon[53210]: pgmap v374: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:26.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:26 smithi008 ceph-mon[66415]: pgmap v375: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:26.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:26 smithi029 ceph-mon[53210]: pgmap v375: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:28.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:28 smithi008 ceph-mon[66415]: pgmap v376: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:28.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:28 smithi029 ceph-mon[53210]: pgmap v376: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:30.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:30 smithi008 ceph-mon[66415]: pgmap v377: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:00:30.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:30 smithi029 ceph-mon[53210]: pgmap v377: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:00:32.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:32 smithi008 ceph-mon[66415]: pgmap v378: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:32.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:32 smithi029 ceph-mon[53210]: pgmap v378: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:34.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:34 smithi008 ceph-mon[66415]: pgmap v379: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:34.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:34 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:00:34.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:34 smithi029 ceph-mon[53210]: pgmap v379: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:34.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:34 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:00:36.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:36 smithi008 ceph-mon[66415]: pgmap v380: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:36.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:36 smithi029 ceph-mon[53210]: pgmap v380: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:38.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:38 smithi008 ceph-mon[66415]: pgmap v381: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:38.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:38 smithi029 ceph-mon[53210]: pgmap v381: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:40.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:40 smithi008 ceph-mon[66415]: pgmap v382: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:00:40.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:40 smithi029 ceph-mon[53210]: pgmap v382: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:00:42.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:42 smithi008 ceph-mon[66415]: pgmap v383: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:42.457 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:42 smithi029 ceph-mon[53210]: pgmap v383: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:44.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:44 smithi029 ceph-mon[53210]: pgmap v384: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:44.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:44 smithi008 ceph-mon[66415]: pgmap v384: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:45.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:45 smithi029 ceph-mon[53210]: pgmap v385: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:45.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:45 smithi008 ceph-mon[66415]: pgmap v385: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:47.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:47 smithi008 ceph-mon[66415]: pgmap v386: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:47.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:47 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:00:47.957 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:47 smithi029 ceph-mon[53210]: pgmap v386: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:47.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:47 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:00:48.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:00:48.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:00:48.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T07:00:48.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T07:00:48.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:00:48.859 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:00:48.859 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:00:48.860 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T07:00:48.860 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T07:00:48.860 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:00:50.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:49 smithi008 ceph-mon[66415]: pgmap v387: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:00:50.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:49 smithi029 ceph-mon[53210]: pgmap v387: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:00:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:51 smithi008 ceph-mon[66415]: pgmap v388: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:52.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:51 smithi029 ceph-mon[53210]: pgmap v388: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:54.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:53 smithi008 ceph-mon[66415]: pgmap v389: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:54.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:53 smithi029 ceph-mon[53210]: pgmap v389: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:56.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:55 smithi008 ceph-mon[66415]: pgmap v390: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:56.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:55 smithi029 ceph-mon[53210]: pgmap v390: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:58.029 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:57 smithi008 ceph-mon[66415]: pgmap v391: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:00:58.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:57 smithi029 ceph-mon[53210]: pgmap v391: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:00.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:00:59 smithi008 ceph-mon[66415]: pgmap v392: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:01:00.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:00:59 smithi029 ceph-mon[53210]: pgmap v392: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:01:02.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:01 smithi008 ceph-mon[66415]: pgmap v393: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:02.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:01 smithi029 ceph-mon[53210]: pgmap v393: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:04.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:03 smithi008 ceph-mon[66415]: pgmap v394: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:04.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:03 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:01:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:03 smithi029 ceph-mon[53210]: pgmap v394: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:04.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:03 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:01:06.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:05 smithi008 ceph-mon[66415]: pgmap v395: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:06.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:05 smithi029 ceph-mon[53210]: pgmap v395: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:08.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:07 smithi029 ceph-mon[53210]: pgmap v396: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:08.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:07 smithi008 ceph-mon[66415]: pgmap v396: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:10.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:09 smithi029 ceph-mon[53210]: pgmap v397: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:01:10.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:09 smithi008 ceph-mon[66415]: pgmap v397: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:01:12.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:11 smithi029 ceph-mon[53210]: pgmap v398: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:12.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:11 smithi008 ceph-mon[66415]: pgmap v398: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:14.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:13 smithi029 ceph-mon[53210]: pgmap v399: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:14.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:13 smithi008 ceph-mon[66415]: pgmap v399: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:16.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:15 smithi029 ceph-mon[53210]: pgmap v400: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:16.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:15 smithi008 ceph-mon[66415]: pgmap v400: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:18.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:17 smithi029 ceph-mon[53210]: pgmap v401: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:18.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:17 smithi008 ceph-mon[66415]: pgmap v401: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:19.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:18 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:01:19.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:18 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:01:20.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:19 smithi029 ceph-mon[53210]: pgmap v402: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:01:20.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:19 smithi008 ceph-mon[66415]: pgmap v402: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:01:22.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:21 smithi008 ceph-mon[66415]: pgmap v403: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:22.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:21 smithi029 ceph-mon[53210]: pgmap v403: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:22.393 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T07:01:23.935 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:23 smithi008 ceph-mon[66415]: pgmap v404: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:24.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:23 smithi029 ceph-mon[53210]: pgmap v404: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:24.494 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T07:01:24.494 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (2m) 102s ago 23m 17.2M - 0.25.0 c8568f914cd2 80540098fb8f 2024-02-10T07:01:24.494 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (9m) 102s ago 23m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 74cc2591e195 2024-02-10T07:01:24.494 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi029 smithi029 running (9m) 2m ago 22m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 834358cbe5df 2024-02-10T07:01:24.494 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (107s) 102s ago 23m 56.7M - 9.4.12 a0ea5e3c46d9 9675a3298934 2024-02-10T07:01:24.495 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.vfwcte smithi008 running (3m) 102s ago 17m 34.4M - 19.0.0-1361-g290212a1 c328ab9dcfc7 6c0c3a230f64 2024-02-10T07:01:24.495 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi029.bwmcyt smithi029 running (3m) 2m ago 17m 14.8M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7c69029ef9bb 2024-02-10T07:01:24.495 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.pxifbt smithi008 *:8443,9283,8765 running (12m) 102s ago 24m 514M - 19.0.0-1361-g290212a1 c328ab9dcfc7 393c10186508 2024-02-10T07:01:24.495 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi029.fuvrtf smithi029 *:8443,9283,8765 running (11m) 2m ago 22m 453M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddecdfad7fbb 2024-02-10T07:01:24.495 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (11m) 102s ago 24m 64.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 ffaeb9e2c721 2024-02-10T07:01:24.495 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi029 smithi029 running (10m) 2m ago 22m 46.7M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 9492552149b2 2024-02-10T07:01:24.495 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.pgujgu smithi008 *:2049 running (3m) 102s ago 16m 72.6M - 5.7 c328ab9dcfc7 00f651c4baff 2024-02-10T07:01:24.495 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi029.cvyqqw smithi029 *:2049 running (3m) 2m ago 16m 71.5M - 5.7 c328ab9dcfc7 f754175aa820 2024-02-10T07:01:24.495 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (2m) 102s ago 23m 8875k - 1.5.0 0da6a335fe13 6fdbf8ce9613 2024-02-10T07:01:24.495 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi029 smithi029 *:9100 running (2m) 2m ago 22m 5440k - 1.5.0 0da6a335fe13 70d1e39d65c1 2024-02-10T07:01:24.495 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (9m) 102s ago 21m 380M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 acb0107eae6c 2024-02-10T07:01:24.495 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (8m) 102s ago 21m 340M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 09363f1c35ae 2024-02-10T07:01:24.495 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (7m) 102s ago 21m 339M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 bd6bbb022085 2024-02-10T07:01:24.495 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (6m) 102s ago 20m 388M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 fc3683d167e9 2024-02-10T07:01:24.495 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi029 running (6m) 2m ago 20m 346M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 2b1f7cbc27ce 2024-02-10T07:01:24.496 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi029 running (5m) 2m ago 19m 310M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 ea4052b4cafa 2024-02-10T07:01:24.496 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi029 running (4m) 2m ago 19m 246M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 d89cf7ba8295 2024-02-10T07:01:24.496 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi029 running (4m) 2m ago 18m 284M 2705M 19.0.0-1361-g290212a1 c328ab9dcfc7 156ce7027d7a 2024-02-10T07:01:24.496 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (2m) 102s ago 22m 47.6M - 2.43.0 a07b618ecd1d faaf1777c75f 2024-02-10T07:01:25.103 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade status' 2024-02-10T07:01:26.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:25 smithi008 ceph-mon[66415]: pgmap v405: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:26.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:25 smithi008 ceph-mon[66415]: from='client.34646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:01:26.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:25 smithi029 ceph-mon[53210]: pgmap v405: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:26.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:25 smithi029 ceph-mon[53210]: from='client.34646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:01:27.208 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:01:27.208 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": null, 2024-02-10T07:01:27.208 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": false, 2024-02-10T07:01:27.208 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "", 2024-02-10T07:01:27.208 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-02-10T07:01:27.208 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": null, 2024-02-10T07:01:27.209 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-02-10T07:01:27.209 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-02-10T07:01:27.209 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:01:27.750 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph health detail' 2024-02-10T07:01:28.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:27 smithi008 ceph-mon[66415]: pgmap v406: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:28.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:27 smithi029 ceph-mon[53210]: pgmap v406: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:29.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:28 smithi008 ceph-mon[66415]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:01:29.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:28 smithi029 ceph-mon[53210]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:01:29.776 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-02-10T07:01:30.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:29 smithi008 ceph-mon[66415]: pgmap v407: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:01:30.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:29 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/1283771682' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:01:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:29 smithi029 ceph-mon[53210]: pgmap v407: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:01:30.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:29 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/1283771682' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T07:01:30.373 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions' 2024-02-10T07:01:31.892 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:31 smithi008 ceph-mon[66415]: pgmap v408: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:32.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:31 smithi029 ceph-mon[53210]: pgmap v408: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:32.388 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:01:32.388 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-02-10T07:01:32.388 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:01:32.388 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:01:32.388 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-02-10T07:01:32.388 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:01:32.388 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:01:32.388 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-02-10T07:01:32.388 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T07:01:32.388 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:01:32.388 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-02-10T07:01:32.389 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T07:01:32.389 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:01:32.389 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-02-10T07:01:32.389 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 14 2024-02-10T07:01:32.389 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:01:32.389 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:01:32.982 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:32 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/104181310' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:01:33.005 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-02-10T07:01:33.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:32 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/104181310' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:01:33.973 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:33 smithi008 ceph-mon[66415]: pgmap v409: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:33.973 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:33 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:01:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:33 smithi029 ceph-mon[53210]: pgmap v409: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:34.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:33 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:01:35.246 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-02-10T07:01:35.825 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-02-10T07:01:36.139 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:35 smithi008 ceph-mon[66415]: pgmap v410: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:36.139 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:35 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/371049181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:01:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:35 smithi029 ceph-mon[53210]: pgmap v410: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:36.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:35 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/371049181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:01:37.919 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)" 2024-02-10T07:01:38.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:37 smithi029 ceph-mon[53210]: pgmap v411: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:38.265 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:37 smithi008 ceph-mon[66415]: pgmap v411: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:38.556 INFO:teuthology.task.sequential:In sequential, running task cephadm.wait_for_service... 2024-02-10T07:01:38.557 INFO:tasks.cephadm:Waiting for ceph service nfs.foo to start (timeout 300)... 2024-02-10T07:01:38.557 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- ceph orch ls -f json 2024-02-10T07:01:39.104 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:38 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/3857421260' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:01:39.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:38 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/3857421260' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T07:01:40.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:39 smithi008 ceph-mon[66415]: pgmap v412: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:01:40.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:39 smithi029 ceph-mon[53210]: pgmap v412: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:01:40.492 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:01:40.493 INFO:teuthology.orchestra.run.smithi008.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-02-10T06:37:19.963271Z", "last_refresh": "2024-02-10T06:59:42.162927Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-02-10T06:37:13.131505Z", "last_refresh": "2024-02-10T06:58:49.780060Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "spec": {"anonymous_access": true, "protocol": "https"}, "status": {"created": "2024-02-10T06:37:16.569177Z", "last_refresh": "2024-02-10T06:59:42.163085Z", "ports": [3000], "running": 1, "size": 1}}, {"placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-02-10T06:44:02.121603Z", "last_refresh": "2024-02-10T06:58:49.780740Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-02-10T06:37:11.800528Z", "last_refresh": "2024-02-10T06:58:49.780168Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi008:172.21.15.8=smithi008", "smithi029:172.21.15.29=smithi029"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-02-10T06:38:03.107535Z", "last_refresh": "2024-02-10T06:58:49.780254Z", "running": 2, "size": 2}}, {"events": ["2024-02-10T07:00:48.079533Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 2049}, "status": {"created": "2024-02-10T06:44:22.979302Z", "last_refresh": "2024-02-10T06:58:49.780819Z", "ports": [2049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-02-10T06:37:18.394994Z", "last_refresh": "2024-02-10T06:58:49.780347Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6", "container_image_name": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", "last_refresh": "2024-02-10T06:58:49.780427Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-02-10T06:37:14.787812Z", "last_refresh": "2024-02-10T06:59:42.163240Z", "ports": [9095], "running": 1, "size": 1}}] 2024-02-10T07:01:41.106 INFO:tasks.cephadm:nfs.foo has 2/2 2024-02-10T07:01:41.106 DEBUG:teuthology.parallel:result is None 2024-02-10T07:01:41.106 INFO:teuthology.run_tasks:Running task vip.exec... 2024-02-10T07:01:41.118 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T07:01:41.118 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'umount /mnt/foo' 2024-02-10T07:01:41.147 INFO:teuthology.orchestra.run.smithi008.stderr:+ umount /mnt/foo 2024-02-10T07:01:41.158 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T07:01:41.168 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T07:01:41.168 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph nfs cluster ls | grep foo' 2024-02-10T07:01:42.182 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:41 smithi008 ceph-mon[66415]: pgmap v413: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:42.182 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:41 smithi008 ceph-mon[66415]: from='client.34670 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:01:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:41 smithi029 ceph-mon[53210]: pgmap v413: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:42.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:41 smithi029 ceph-mon[53210]: from='client.34670 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:01:43.181 INFO:teuthology.orchestra.run.smithi008.stdout: "foo" 2024-02-10T07:01:43.829 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- bash -c 'ceph nfs export ls foo --detailed' 2024-02-10T07:01:44.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:43 smithi008 ceph-mon[66415]: pgmap v414: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:44.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:43 smithi029 ceph-mon[53210]: pgmap v414: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:45.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:44 smithi008 ceph-mon[66415]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:01:45.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:44 smithi029 ceph-mon[53210]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:01:45.808 INFO:teuthology.orchestra.run.smithi008.stdout:[ 2024-02-10T07:01:45.808 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:01:45.808 INFO:teuthology.orchestra.run.smithi008.stdout: "access_type": "RW", 2024-02-10T07:01:45.808 INFO:teuthology.orchestra.run.smithi008.stdout: "clients": [], 2024-02-10T07:01:45.808 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_id": "foo", 2024-02-10T07:01:45.808 INFO:teuthology.orchestra.run.smithi008.stdout: "export_id": 1, 2024-02-10T07:01:45.808 INFO:teuthology.orchestra.run.smithi008.stdout: "fsal": { 2024-02-10T07:01:45.808 INFO:teuthology.orchestra.run.smithi008.stdout: "fs_name": "foofs", 2024-02-10T07:01:45.808 INFO:teuthology.orchestra.run.smithi008.stdout: "name": "CEPH", 2024-02-10T07:01:45.808 INFO:teuthology.orchestra.run.smithi008.stdout: "user_id": "nfs.foo.1" 2024-02-10T07:01:45.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:01:45.809 INFO:teuthology.orchestra.run.smithi008.stdout: "path": "/", 2024-02-10T07:01:45.809 INFO:teuthology.orchestra.run.smithi008.stdout: "protocols": [ 2024-02-10T07:01:45.809 INFO:teuthology.orchestra.run.smithi008.stdout: 4 2024-02-10T07:01:45.809 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:01:45.809 INFO:teuthology.orchestra.run.smithi008.stdout: "pseudo": "/fake", 2024-02-10T07:01:45.809 INFO:teuthology.orchestra.run.smithi008.stdout: "security_label": true, 2024-02-10T07:01:45.809 INFO:teuthology.orchestra.run.smithi008.stdout: "squash": "none", 2024-02-10T07:01:45.809 INFO:teuthology.orchestra.run.smithi008.stdout: "transports": [ 2024-02-10T07:01:45.809 INFO:teuthology.orchestra.run.smithi008.stdout: "TCP" 2024-02-10T07:01:45.809 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:01:45.809 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:01:45.809 INFO:teuthology.orchestra.run.smithi008.stdout:] 2024-02-10T07:01:46.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:45 smithi029 ceph-mon[53210]: pgmap v415: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:46.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:45 smithi008 ceph-mon[66415]: pgmap v415: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:46.529 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- bash -c 'rados -p .nfs --all ls -' 2024-02-10T07:01:47.174 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:46 smithi008 ceph-mon[66415]: from='client.34678 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:01:47.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:46 smithi029 ceph-mon[53210]: from='client.34678 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:01:48.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:47 smithi008 ceph-mon[66415]: pgmap v416: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:48.127 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:47 smithi029 ceph-mon[53210]: pgmap v416: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T07:01:48.274 INFO:teuthology.orchestra.run.smithi008.stdout:foo rec-0000000000000004:nfs.foo.1 2024-02-10T07:01:48.274 INFO:teuthology.orchestra.run.smithi008.stdout:foo grace 2024-02-10T07:01:48.274 INFO:teuthology.orchestra.run.smithi008.stdout:foo rec-0000000000000004:nfs.foo.0 2024-02-10T07:01:48.274 INFO:teuthology.orchestra.run.smithi008.stdout:foo conf-nfs.foo 2024-02-10T07:01:48.274 INFO:teuthology.orchestra.run.smithi008.stdout:foo export-1 2024-02-10T07:01:48.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:48 smithi008 ceph-mon[66415]: mgrmap e42: smithi008.pxifbt(active, since 11m), standbys: smithi029.fuvrtf 2024-02-10T07:01:48.961 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:01:48.962 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:48 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:01:48.983 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T07:01:48.993 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T07:01:48.994 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 -- bash -c 'set -ex 2024-02-10T07:01:48.994 DEBUG:teuthology.orchestra.run.smithi008:> [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] 2024-02-10T07:01:48.994 DEBUG:teuthology.orchestra.run.smithi008:> ' 2024-02-10T07:01:49.207 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:48 smithi029 ceph-mon[53210]: mgrmap e42: smithi008.pxifbt(active, since 11m), standbys: smithi029.fuvrtf 2024-02-10T07:01:49.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:01:49.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:48 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:01:50.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:49 smithi029 ceph-mon[53210]: pgmap v417: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:01:50.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:01:50.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:01:50.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T07:01:50.208 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:49 smithi029 ceph-mon[53210]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T07:01:50.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:49 smithi008 ceph-mon[66415]: pgmap v417: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:01:50.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:01:50.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:01:50.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T07:01:50.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:49 smithi008 ceph-mon[66415]: from='mgr.24627 172.21.15.8:0/2829354584' entity='mgr.smithi008.pxifbt' 2024-02-10T07:01:50.814 INFO:teuthology.orchestra.run.smithi008.stderr:++ ceph config get mgr mgr/cephadm/migration_current 2024-02-10T07:01:51.056 INFO:teuthology.orchestra.run.smithi008.stderr:+ [[ 6 -gt 2 ]] 2024-02-10T07:01:51.737 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-10T07:01:51.748 INFO:tasks.cephadm:Teardown begin 2024-02-10T07:01:51.749 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:01:51.780 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:01:51.816 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-10T07:01:51.816 DEBUG:teuthology.orchestra.run.smithi008:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T07:01:51.836 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T07:01:51.874 INFO:tasks.cephadm:Stopping all daemons... 2024-02-10T07:01:51.874 INFO:tasks.cephadm.mon.smithi008:Stopping mon.smithi008... 2024-02-10T07:01:51.874 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mon.smithi008 2024-02-10T07:01:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:51 smithi008 ceph-mon[66415]: pgmap v418: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T07:01:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:51 smithi008 ceph-mon[66415]: from='client.? 172.21.15.8:0/1974431968' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-02-10T07:01:52.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:51 smithi008 systemd[1]: Stopping Ceph mon.smithi008 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T07:01:52.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:51 smithi029 ceph-mon[53210]: pgmap v418: 97 pgs: 97 active+clean; 234 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T07:01:52.209 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:51 smithi029 ceph-mon[53210]: from='client.? 172.21.15.8:0/1974431968' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-02-10T07:01:52.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:52 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008[66411]: 2024-02-10T07:01:52.255+0000 7f8c91c47700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi008 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:01:52.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:52 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008[66411]: 2024-02-10T07:01:52.255+0000 7f8c91c47700 -1 mon.smithi008@0(leader) e3 *** Got Signal Terminated *** 2024-02-10T07:01:52.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:52 smithi008 podman[112107]: 2024-02-10 07:01:52.429281253 +0000 UTC m=+0.360135999 container died ffaeb9e2c72103a9362c6dc13c899629eac8502f690ae3e5add304a1009dc0ed (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:01:53.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:52 smithi008 podman[112107]: 2024-02-10 07:01:52.797319691 +0000 UTC m=+0.728174437 container cleanup ffaeb9e2c72103a9362c6dc13c899629eac8502f690ae3e5add304a1009dc0ed (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-10T07:01:53.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Feb 10 07:01:52 smithi008 bash[112107]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi008 2024-02-10T07:01:53.233 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mon.smithi008.service' 2024-02-10T07:01:53.284 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:01:53.285 INFO:tasks.cephadm.mon.smithi008:Stopped mon.smithi008 2024-02-10T07:01:53.285 INFO:tasks.cephadm.mon.smithi029:Stopping mon.smithi029... 2024-02-10T07:01:53.285 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mon.smithi029 2024-02-10T07:01:53.623 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:53 smithi029 systemd[1]: Stopping Ceph mon.smithi029 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T07:01:53.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:53 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi029[53206]: 2024-02-10T07:01:53.621+0000 7ff797359700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi029 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:01:53.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:53 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi029[53206]: 2024-02-10T07:01:53.621+0000 7ff797359700 -1 mon.smithi029@1(peon) e3 *** Got Signal Terminated *** 2024-02-10T07:01:53.958 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:53 smithi029 podman[78875]: 2024-02-10 07:01:53.775603489 +0000 UTC m=+0.292837627 container died 9492552149b2e4b58d8a07a94a69adfc7dce601716d19403d34b90b61a5a14ba (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi029, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, ceph=True) 2024-02-10T07:01:54.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:54 smithi029 podman[78875]: 2024-02-10 07:01:54.319377097 +0000 UTC m=+0.836611289 container cleanup 9492552149b2e4b58d8a07a94a69adfc7dce601716d19403d34b90b61a5a14ba (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi029, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, RELEASE=main-290212a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-02-10T07:01:54.458 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:54 smithi029 bash[78875]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi029 2024-02-10T07:01:54.885 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Feb 10 07:01:54 smithi029 podman[78889]: 2024-02-10 07:01:54.631879676 +0000 UTC m=+0.853579367 container remove 9492552149b2e4b58d8a07a94a69adfc7dce601716d19403d34b90b61a5a14ba (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-mon-smithi029, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-10T07:01:54.909 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@mon.smithi029.service' 2024-02-10T07:01:54.963 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:01:54.964 INFO:tasks.cephadm.mon.smithi029:Stopped mon.smithi029 2024-02-10T07:01:54.964 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-10T07:01:54.964 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.0 2024-02-10T07:01:55.283 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:01:55 smithi008 systemd[1]: Stopping Ceph osd.0 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T07:01:55.783 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:01:55 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0[72537]: 2024-02-10T07:01:55.293+0000 7fda7f7c7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:01:55.783 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:01:55 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0[72537]: 2024-02-10T07:01:55.293+0000 7fda7f7c7700 -1 osd.0 142 *** Got signal Terminated *** 2024-02-10T07:01:55.783 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:01:55 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0[72537]: 2024-02-10T07:01:55.293+0000 7fda7f7c7700 -1 osd.0 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:02:01.095 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:02:00 smithi008 podman[112269]: 2024-02-10 07:02:00.783558871 +0000 UTC m=+5.624070150 container died acb0107eae6cd46a6db146051796c3e47490b8c69574663279ec5aa184d5440f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.license=GPLv2) 2024-02-10T07:02:01.445 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:02:01 smithi008 podman[112269]: 2024-02-10 07:02:01.18525454 +0000 UTC m=+6.025765811 container cleanup acb0107eae6cd46a6db146051796c3e47490b8c69574663279ec5aa184d5440f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-290212a, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:02:01.445 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:02:01 smithi008 bash[112269]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0 2024-02-10T07:02:01.783 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:02:01 smithi008 podman[112287]: 2024-02-10 07:02:01.44519384 +0000 UTC m=+0.661002147 container remove acb0107eae6cd46a6db146051796c3e47490b8c69574663279ec5aa184d5440f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-10T07:02:02.084 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:02:01 smithi008 podman[112380]: 2024-02-10 07:02:01.705604046 +0000 UTC m=+0.018981095 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:02:02.084 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:02:01 smithi008 podman[112380]: 2024-02-10 07:02:01.829552229 +0000 UTC m=+0.142929271 container create d6655bd76cf3f52dbebda95f9ac77cdd85e100d65e6ddb918d1e6333354bca38 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, RELEASE=main-290212a, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=) 2024-02-10T07:02:02.399 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:02:02 smithi008 podman[112380]: 2024-02-10 07:02:02.083022736 +0000 UTC m=+0.396399778 container init d6655bd76cf3f52dbebda95f9ac77cdd85e100d65e6ddb918d1e6333354bca38 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-290212a) 2024-02-10T07:02:02.477 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:02:02 smithi008 podman[112380]: 2024-02-10 07:02:02.086518668 +0000 UTC m=+0.399895720 container start d6655bd76cf3f52dbebda95f9ac77cdd85e100d65e6ddb918d1e6333354bca38 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-290212a, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=) 2024-02-10T07:02:02.477 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:02:02 smithi008 podman[112380]: 2024-02-10 07:02:02.166390267 +0000 UTC m=+0.479767312 container attach d6655bd76cf3f52dbebda95f9ac77cdd85e100d65e6ddb918d1e6333354bca38 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, io.buildah.version=1.33.2) 2024-02-10T07:02:02.737 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:02:02 smithi008 podman[112380]: 2024-02-10 07:02:02.399620532 +0000 UTC m=+0.712997580 container died d6655bd76cf3f52dbebda95f9ac77cdd85e100d65e6ddb918d1e6333354bca38 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:02:03.025 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.0.service' 2024-02-10T07:02:03.068 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:02:03.068 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-10T07:02:03.068 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-10T07:02:03.068 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.1 2024-02-10T07:02:03.283 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:02:03 smithi008 systemd[1]: Stopping Ceph osd.1 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T07:02:03.783 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:02:03 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1[75212]: 2024-02-10T07:02:03.414+0000 7f80a5db2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:02:03.783 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:02:03 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1[75212]: 2024-02-10T07:02:03.414+0000 7f80a5db2700 -1 osd.1 142 *** Got signal Terminated *** 2024-02-10T07:02:03.783 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:02:03 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1[75212]: 2024-02-10T07:02:03.414+0000 7f80a5db2700 -1 osd.1 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:02:09.033 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:02:08 smithi008 podman[112488]: 2024-02-10 07:02:08.788726463 +0000 UTC m=+5.494160957 container died 09363f1c35aece938da7200cdb49d8ff866039e78ec9cc267438a974cad0ae4e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_CLEAN=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2) 2024-02-10T07:02:09.424 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:02:09 smithi008 podman[112488]: 2024-02-10 07:02:09.165164928 +0000 UTC m=+5.870599416 container cleanup 09363f1c35aece938da7200cdb49d8ff866039e78ec9cc267438a974cad0ae4e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-290212a, io.buildah.version=1.33.2) 2024-02-10T07:02:09.425 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:02:09 smithi008 bash[112488]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1 2024-02-10T07:02:09.783 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:02:09 smithi008 podman[112514]: 2024-02-10 07:02:09.424923027 +0000 UTC m=+0.634834900 container remove 09363f1c35aece938da7200cdb49d8ff866039e78ec9cc267438a974cad0ae4e (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux ) 2024-02-10T07:02:10.104 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:02:09 smithi008 podman[112611]: 2024-02-10 07:02:09.784196801 +0000 UTC m=+0.114405773 container create 8a3bbe40c1eede6924001f69e4054302055679238bdff5c9fea5612842fc540f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2) 2024-02-10T07:02:10.104 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:02:09 smithi008 podman[112611]: 2024-02-10 07:02:09.688283167 +0000 UTC m=+0.018492152 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:02:10.104 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:02:10 smithi008 podman[112611]: 2024-02-10 07:02:10.005551999 +0000 UTC m=+0.335760975 container init 8a3bbe40c1eede6924001f69e4054302055679238bdff5c9fea5612842fc540f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-10T07:02:10.104 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:02:10 smithi008 podman[112611]: 2024-02-10 07:02:10.00880082 +0000 UTC m=+0.339009796 container start 8a3bbe40c1eede6924001f69e4054302055679238bdff5c9fea5612842fc540f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.license=GPLv2, RELEASE=main-290212a, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-10T07:02:10.104 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:02:10 smithi008 podman[112611]: 2024-02-10 07:02:10.10380693 +0000 UTC m=+0.434015903 container attach 8a3bbe40c1eede6924001f69e4054302055679238bdff5c9fea5612842fc540f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_CLEAN=True, ceph=True) 2024-02-10T07:02:10.533 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:02:10 smithi008 podman[112611]: 2024-02-10 07:02:10.309896628 +0000 UTC m=+0.640105601 container died 8a3bbe40c1eede6924001f69e4054302055679238bdff5c9fea5612842fc540f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD) 2024-02-10T07:02:10.957 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.1.service' 2024-02-10T07:02:11.005 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:02:11.005 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-10T07:02:11.006 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-10T07:02:11.006 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.2 2024-02-10T07:02:11.284 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:02:11 smithi008 systemd[1]: Stopping Ceph osd.2 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T07:02:11.783 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:02:11 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2[77815]: 2024-02-10T07:02:11.322+0000 7f61afb86700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:02:11.783 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:02:11 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2[77815]: 2024-02-10T07:02:11.322+0000 7f61afb86700 -1 osd.2 142 *** Got signal Terminated *** 2024-02-10T07:02:11.783 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:02:11 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2[77815]: 2024-02-10T07:02:11.322+0000 7f61afb86700 -1 osd.2 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:02:17.071 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:02:16 smithi008 podman[112715]: 2024-02-10 07:02:16.788829499 +0000 UTC m=+5.593152984 container died bd6bbb0220855bf9c1e89cadee5fab7c3a325d0fd768e4ec5b0d8910ab7a1e05 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-02-10T07:02:17.437 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:02:17 smithi008 podman[112715]: 2024-02-10 07:02:17.169634174 +0000 UTC m=+5.973957668 container cleanup bd6bbb0220855bf9c1e89cadee5fab7c3a325d0fd768e4ec5b0d8910ab7a1e05 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.license=GPLv2) 2024-02-10T07:02:17.437 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:02:17 smithi008 bash[112715]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2 2024-02-10T07:02:17.783 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:02:17 smithi008 podman[112764]: 2024-02-10 07:02:17.437651223 +0000 UTC m=+0.652416117 container remove bd6bbb0220855bf9c1e89cadee5fab7c3a325d0fd768e4ec5b0d8910ab7a1e05 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2, GIT_CLEAN=True, io.buildah.version=1.33.2, RELEASE=main-290212a, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:02:18.133 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:02:17 smithi008 podman[112854]: 2024-02-10 07:02:17.690657572 +0000 UTC m=+0.018011848 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:02:18.133 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:02:17 smithi008 podman[112854]: 2024-02-10 07:02:17.813973007 +0000 UTC m=+0.141327318 container create 412d1a740e79fbfbd3ea5372595695097cb0da541fd1f3b5d7999b1a52a0dbb7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-10T07:02:18.133 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:02:18 smithi008 podman[112854]: 2024-02-10 07:02:18.033778294 +0000 UTC m=+0.361132568 container init 412d1a740e79fbfbd3ea5372595695097cb0da541fd1f3b5d7999b1a52a0dbb7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, ceph=True) 2024-02-10T07:02:18.133 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:02:18 smithi008 podman[112854]: 2024-02-10 07:02:18.037089517 +0000 UTC m=+0.364443794 container start 412d1a740e79fbfbd3ea5372595695097cb0da541fd1f3b5d7999b1a52a0dbb7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:02:18.134 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:02:18 smithi008 podman[112854]: 2024-02-10 07:02:18.133273342 +0000 UTC m=+0.460627618 container attach 412d1a740e79fbfbd3ea5372595695097cb0da541fd1f3b5d7999b1a52a0dbb7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-290212a) 2024-02-10T07:02:18.533 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:02:18 smithi008 podman[112854]: 2024-02-10 07:02:18.331173543 +0000 UTC m=+0.658527818 container died 412d1a740e79fbfbd3ea5372595695097cb0da541fd1f3b5d7999b1a52a0dbb7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-2-deactivate, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-10T07:02:18.888 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.2.service' 2024-02-10T07:02:18.931 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:02:18.931 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-10T07:02:18.931 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-10T07:02:18.931 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.3 2024-02-10T07:02:19.265 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:02:19 smithi008 systemd[1]: Stopping Ceph osd.3 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T07:02:19.533 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:02:19 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3[80539]: 2024-02-10T07:02:19.264+0000 7f5783504700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:02:19.533 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:02:19 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3[80539]: 2024-02-10T07:02:19.264+0000 7f5783504700 -1 osd.3 142 *** Got signal Terminated *** 2024-02-10T07:02:19.533 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:02:19 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3[80539]: 2024-02-10T07:02:19.264+0000 7f5783504700 -1 osd.3 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:02:22.208 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:21 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:21.801+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:22.708 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:22 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:22.323+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:23.035 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:02:22 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3[80539]: 2024-02-10T07:02:22.770+0000 7f577baf0700 -1 osd.3 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:57.034294+0000 front 2024-02-10T07:01:57.034415+0000 (oldest deadline 2024-02-10T07:02:22.334345+0000) 2024-02-10T07:02:23.208 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:23 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T07:02:23.050+0000 7fa557443700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:57.750685+0000 front 2024-02-10T07:01:57.750665+0000 (oldest deadline 2024-02-10T07:02:22.450794+0000) 2024-02-10T07:02:23.208 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:22 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:22.804+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:23.632 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:23 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:23.312+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:23.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:23 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:23.805+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:24.033 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:02:23 smithi008 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3[80539]: 2024-02-10T07:02:23.741+0000 7f577baf0700 -1 osd.3 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:57.034294+0000 front 2024-02-10T07:01:57.034415+0000 (oldest deadline 2024-02-10T07:02:22.334345+0000) 2024-02-10T07:02:24.262 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:24 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T07:02:24.009+0000 7fa557443700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:57.750685+0000 front 2024-02-10T07:01:57.750665+0000 (oldest deadline 2024-02-10T07:02:22.450794+0000) 2024-02-10T07:02:24.708 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:24 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:24.327+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:24.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:24 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:24.261+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:24.941 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:02:24 smithi008 podman[112960]: 2024-02-10 07:02:24.658738439 +0000 UTC m=+5.506247800 container died fc3683d167e9a87718b94f71357c557e4e1a2bd99891e0c957902f35c2a4ef36 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-10T07:02:25.208 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:24 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:24.757+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:25.208 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:24 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T07:02:24.971+0000 7fa557443700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:57.750685+0000 front 2024-02-10T07:01:57.750665+0000 (oldest deadline 2024-02-10T07:02:22.450794+0000) 2024-02-10T07:02:25.266 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:02:25 smithi008 podman[112960]: 2024-02-10 07:02:25.023140923 +0000 UTC m=+5.870650284 container cleanup fc3683d167e9a87718b94f71357c557e4e1a2bd99891e0c957902f35c2a4ef36 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-02-10T07:02:25.266 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:02:25 smithi008 bash[112960]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3 2024-02-10T07:02:25.533 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:02:25 smithi008 podman[112979]: 2024-02-10 07:02:25.26616724 +0000 UTC m=+0.609115377 container remove fc3683d167e9a87718b94f71357c557e4e1a2bd99891e0c957902f35c2a4ef36 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a) 2024-02-10T07:02:25.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:25 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:25.268+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:25.708 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:25 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:25.296+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:25.869 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:02:25 smithi008 podman[113074]: 2024-02-10 07:02:25.575308463 +0000 UTC m=+0.112398368 container create 5394a5c09703e9cbca59ae125388be9453a4c07bea4a870246007c1ed31a8a6d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-deactivate, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:02:25.869 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:02:25 smithi008 podman[113074]: 2024-02-10 07:02:25.481002255 +0000 UTC m=+0.018092173 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:02:25.870 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:02:25 smithi008 podman[113074]: 2024-02-10 07:02:25.778109078 +0000 UTC m=+0.315198984 container init 5394a5c09703e9cbca59ae125388be9453a4c07bea4a870246007c1ed31a8a6d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-10T07:02:25.870 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:02:25 smithi008 podman[113074]: 2024-02-10 07:02:25.781251753 +0000 UTC m=+0.318341662 container start 5394a5c09703e9cbca59ae125388be9453a4c07bea4a870246007c1ed31a8a6d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240131) 2024-02-10T07:02:25.870 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:02:25 smithi008 podman[113074]: 2024-02-10 07:02:25.869693536 +0000 UTC m=+0.406783440 container attach 5394a5c09703e9cbca59ae125388be9453a4c07bea4a870246007c1ed31a8a6d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-10T07:02:26.011 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:25 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:25.714+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:26.283 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:02:26 smithi008 podman[113074]: 2024-02-10 07:02:26.08930316 +0000 UTC m=+0.626393069 container died 5394a5c09703e9cbca59ae125388be9453a4c07bea4a870246007c1ed31a8a6d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-3-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-10T07:02:26.338 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:26 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T07:02:26.008+0000 7fa557443700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:57.750685+0000 front 2024-02-10T07:01:57.750665+0000 (oldest deadline 2024-02-10T07:02:22.450794+0000) 2024-02-10T07:02:26.339 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:26 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:26.238+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:26.675 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.3.service' 2024-02-10T07:02:26.707 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:26 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:26.338+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:26.714 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:02:26.714 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-10T07:02:26.714 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-10T07:02:26.715 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.4 2024-02-10T07:02:26.981 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:26 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:26.722+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:26.981 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:26 smithi029 systemd[1]: Stopping Ceph osd.4 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T07:02:26.981 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:26 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T07:02:26.979+0000 7fa557443700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:57.750685+0000 front 2024-02-10T07:01:57.750665+0000 (oldest deadline 2024-02-10T07:02:22.450794+0000) 2024-02-10T07:02:27.264 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:27 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T07:02:27.036+0000 7fa55ee57700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:02:27.264 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:27 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T07:02:27.036+0000 7fa55ee57700 -1 osd.4 142 *** Got signal Terminated *** 2024-02-10T07:02:27.264 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:27 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T07:02:27.036+0000 7fa55ee57700 -1 osd.4 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:02:27.678 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:27 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:27.263+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:27.678 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:27 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:27.347+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:27.957 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:27 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:27.677+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:28.299 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:27 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T07:02:27.970+0000 7fa557443700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:57.750685+0000 front 2024-02-10T07:01:57.750665+0000 (oldest deadline 2024-02-10T07:02:22.450794+0000) 2024-02-10T07:02:28.678 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:28 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:28.299+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:28.679 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:28 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:28.385+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:28.957 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:28 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:28.678+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:29.343 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:29 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T07:02:29.011+0000 7fa557443700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:57.750685+0000 front 2024-02-10T07:01:57.750665+0000 (oldest deadline 2024-02-10T07:02:22.450794+0000) 2024-02-10T07:02:29.669 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:29 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:29.403+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:29.669 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:29 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:29.342+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:29.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:29 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:29.669+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:30.361 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:30 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T07:02:30.040+0000 7fa557443700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:57.750685+0000 front 2024-02-10T07:01:57.750665+0000 (oldest deadline 2024-02-10T07:02:22.450794+0000) 2024-02-10T07:02:30.682 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:30 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:30.360+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:30.682 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:30 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:30.388+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:30.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:30 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:30.681+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:31.349 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:31 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T07:02:31.004+0000 7fa557443700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:57.750685+0000 front 2024-02-10T07:01:57.750665+0000 (oldest deadline 2024-02-10T07:02:22.450794+0000) 2024-02-10T07:02:31.350 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:31 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T07:02:31.004+0000 7fa557443700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.751828+0000 front 2024-02-10T07:02:04.751925+0000 (oldest deadline 2024-02-10T07:02:30.051899+0000) 2024-02-10T07:02:31.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:31 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:31.349+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:31.708 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:31 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:31.413+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:31.708 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:31 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:31.413+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:06.736650+0000 front 2024-02-10T07:02:06.736719+0000 (oldest deadline 2024-02-10T07:02:30.836641+0000) 2024-02-10T07:02:31.986 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:31 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:31.717+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:32.378 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:31 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T07:02:31.985+0000 7fa557443700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:57.750685+0000 front 2024-02-10T07:01:57.750665+0000 (oldest deadline 2024-02-10T07:02:22.450794+0000) 2024-02-10T07:02:32.378 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:31 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4[58139]: 2024-02-10T07:02:31.985+0000 7fa557443700 -1 osd.4 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.751828+0000 front 2024-02-10T07:02:04.751925+0000 (oldest deadline 2024-02-10T07:02:30.051899+0000) 2024-02-10T07:02:32.697 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:32 smithi029 podman[79042]: 2024-02-10 07:02:32.474370993 +0000 UTC m=+5.555891726 container died 2b1f7cbc27ce22de862cb2cb319f02b0da2e1695516a5e6f6bdb7f4f12e489d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2) 2024-02-10T07:02:32.698 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:32 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:32.447+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:32.698 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:32 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:32.447+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:06.736650+0000 front 2024-02-10T07:02:06.736719+0000 (oldest deadline 2024-02-10T07:02:30.836641+0000) 2024-02-10T07:02:32.698 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:32 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:32.377+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:32.698 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:32 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:32.377+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:32.957 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:32 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:32.697+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:33.351 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:32 smithi029 podman[79042]: 2024-02-10 07:02:32.998582966 +0000 UTC m=+6.080103742 container cleanup 2b1f7cbc27ce22de862cb2cb319f02b0da2e1695516a5e6f6bdb7f4f12e489d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4, ceph=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-10T07:02:33.351 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:32 smithi029 bash[79042]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4 2024-02-10T07:02:33.351 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:33 smithi029 podman[79061]: 2024-02-10 07:02:33.244611341 +0000 UTC m=+0.772422228 container remove 2b1f7cbc27ce22de862cb2cb319f02b0da2e1695516a5e6f6bdb7f4f12e489d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:02:33.676 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:33 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:33.348+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:33.676 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:33 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:33.348+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:33.676 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:33 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:33.419+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:33.676 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:33 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:33.419+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:06.736650+0000 front 2024-02-10T07:02:06.736719+0000 (oldest deadline 2024-02-10T07:02:30.836641+0000) 2024-02-10T07:02:33.944 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:33 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:33.693+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:33.944 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:33 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:33.693+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.944306+0000 front 2024-02-10T07:02:04.944329+0000 (oldest deadline 2024-02-10T07:02:33.144779+0000) 2024-02-10T07:02:33.944 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:33 smithi029 podman[79188]: 2024-02-10 07:02:33.57883124 +0000 UTC m=+0.017863545 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:02:33.944 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:33 smithi029 podman[79188]: 2024-02-10 07:02:33.688032106 +0000 UTC m=+0.127064401 container create c0184fab1c9c4d12567d0f20d9172ddb814d34f4aeea069c63c72ece1798a13b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-290212a) 2024-02-10T07:02:33.944 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:33 smithi029 podman[79188]: 2024-02-10 07:02:33.944375405 +0000 UTC m=+0.383407703 container init c0184fab1c9c4d12567d0f20d9172ddb814d34f4aeea069c63c72ece1798a13b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, RELEASE=main-290212a, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-10T07:02:34.208 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:33 smithi029 podman[79188]: 2024-02-10 07:02:33.947693071 +0000 UTC m=+0.386725367 container start c0184fab1c9c4d12567d0f20d9172ddb814d34f4aeea069c63c72ece1798a13b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-10T07:02:34.208 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:34 smithi029 podman[79188]: 2024-02-10 07:02:34.103543334 +0000 UTC m=+0.542575656 container attach c0184fab1c9c4d12567d0f20d9172ddb814d34f4aeea069c63c72ece1798a13b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, RELEASE=main-290212a, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2) 2024-02-10T07:02:34.670 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:34 smithi029 podman[79188]: 2024-02-10 07:02:34.24513815 +0000 UTC m=+0.684170448 container died c0184fab1c9c4d12567d0f20d9172ddb814d34f4aeea069c63c72ece1798a13b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:02:34.671 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:34 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:34.465+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:34.671 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:34 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:34.465+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:06.736650+0000 front 2024-02-10T07:02:06.736719+0000 (oldest deadline 2024-02-10T07:02:30.836641+0000) 2024-02-10T07:02:34.671 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:34 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:34.395+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:34.671 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:34 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:34.395+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:34.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.4.service' 2024-02-10T07:02:34.923 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:34 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:34.670+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:34.923 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:34 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:34.670+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.944306+0000 front 2024-02-10T07:02:04.944329+0000 (oldest deadline 2024-02-10T07:02:33.144779+0000) 2024-02-10T07:02:34.924 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:34 smithi029 podman[79214]: 2024-02-10 07:02:34.870725213 +0000 UTC m=+0.620387438 container remove c0184fab1c9c4d12567d0f20d9172ddb814d34f4aeea069c63c72ece1798a13b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-4-deactivate, RELEASE=main-290212a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:02:34.924 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:34 smithi029 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.4.service: Deactivated successfully. 2024-02-10T07:02:34.924 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:34 smithi029 systemd[1]: Stopped Ceph osd.4 for a5f941ec-c7de-11ee-95b8-87774f69a715. 2024-02-10T07:02:34.924 INFO:journalctl@ceph.osd.4.smithi029.stdout:Feb 10 07:02:34 smithi029 systemd[1]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.4.service: Consumed 52.871s CPU time. 2024-02-10T07:02:34.939 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:02:34.939 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-10T07:02:34.939 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-10T07:02:34.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.5 2024-02-10T07:02:35.208 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:35 smithi029 systemd[1]: Stopping Ceph osd.5 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T07:02:35.642 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:35 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:35.431+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:35.642 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:35 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:35.431+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:35.643 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:35 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:35.274+0000 7fb52d867700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:02:35.643 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:35 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:35.274+0000 7fb52d867700 -1 osd.5 142 *** Got signal Terminated *** 2024-02-10T07:02:35.643 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:35 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:35.274+0000 7fb52d867700 -1 osd.5 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:02:35.643 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:35 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:35.506+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:35.643 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:35 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:35.506+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:06.736650+0000 front 2024-02-10T07:02:06.736719+0000 (oldest deadline 2024-02-10T07:02:30.836641+0000) 2024-02-10T07:02:35.957 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:35 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:35.641+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:35.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:35 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:35.641+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.944306+0000 front 2024-02-10T07:02:04.944329+0000 (oldest deadline 2024-02-10T07:02:33.144779+0000) 2024-02-10T07:02:36.708 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:36 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:36.540+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:36.708 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:36 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:36.540+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:06.736650+0000 front 2024-02-10T07:02:06.736719+0000 (oldest deadline 2024-02-10T07:02:30.836641+0000) 2024-02-10T07:02:36.708 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:36 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:36.648+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:36.708 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:36 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:36.648+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.944306+0000 front 2024-02-10T07:02:04.944329+0000 (oldest deadline 2024-02-10T07:02:33.144779+0000) 2024-02-10T07:02:36.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:36 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:36.400+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:36.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:36 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:36.400+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:37.662 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:37 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:37.539+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:37.662 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:37 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:37.539+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:06.736650+0000 front 2024-02-10T07:02:06.736719+0000 (oldest deadline 2024-02-10T07:02:30.836641+0000) 2024-02-10T07:02:37.662 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:37 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:37.368+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:37.663 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:37 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:37.368+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:37.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:37 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:37.661+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:37.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:37 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:37.661+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.944306+0000 front 2024-02-10T07:02:04.944329+0000 (oldest deadline 2024-02-10T07:02:33.144779+0000) 2024-02-10T07:02:38.661 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:38 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:38.506+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:38.662 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:38 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:38.506+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:06.736650+0000 front 2024-02-10T07:02:06.736719+0000 (oldest deadline 2024-02-10T07:02:30.836641+0000) 2024-02-10T07:02:38.662 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:38 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:38.406+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:38.662 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:38 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:38.406+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:38.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:38 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:38.661+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:38.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:38 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:38.661+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.944306+0000 front 2024-02-10T07:02:04.944329+0000 (oldest deadline 2024-02-10T07:02:33.144779+0000) 2024-02-10T07:02:39.702 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:39 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:39.521+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:56.235109+0000 front 2024-02-10T07:01:56.235223+0000 (oldest deadline 2024-02-10T07:02:21.435511+0000) 2024-02-10T07:02:39.702 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:39 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5[60382]: 2024-02-10T07:02:39.521+0000 7fb525e53700 -1 osd.5 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:06.736650+0000 front 2024-02-10T07:02:06.736719+0000 (oldest deadline 2024-02-10T07:02:30.836641+0000) 2024-02-10T07:02:39.703 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:39 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:39.424+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:39.703 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:39 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:39.424+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:39.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:39 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:39.701+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:39.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:39 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:39.701+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.944306+0000 front 2024-02-10T07:02:04.944329+0000 (oldest deadline 2024-02-10T07:02:33.144779+0000) 2024-02-10T07:02:39.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:39 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:39.701+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:14.245758+0000 front 2024-02-10T07:02:14.245691+0000 (oldest deadline 2024-02-10T07:02:38.945657+0000) 2024-02-10T07:02:40.723 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:40 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:40.470+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:40.723 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:40 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:40.470+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:40.723 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:40 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:40.719+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:40.723 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:40 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:40.719+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.944306+0000 front 2024-02-10T07:02:04.944329+0000 (oldest deadline 2024-02-10T07:02:33.144779+0000) 2024-02-10T07:02:40.723 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:40 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:40.719+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:14.245758+0000 front 2024-02-10T07:02:14.245691+0000 (oldest deadline 2024-02-10T07:02:38.945657+0000) 2024-02-10T07:02:41.019 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:40 smithi029 podman[79294]: 2024-02-10 07:02:40.723134029 +0000 UTC m=+5.564165216 container died ea4052b4cafaafbfa0cc5018e0d297731b2f70e782a7df7ca641843e67be91ad (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5, RELEASE=main-290212a, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-10T07:02:41.406 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:41 smithi029 podman[79294]: 2024-02-10 07:02:41.13524547 +0000 UTC m=+5.976276651 container cleanup ea4052b4cafaafbfa0cc5018e0d297731b2f70e782a7df7ca641843e67be91ad (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.build-date=20240131) 2024-02-10T07:02:41.406 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:41 smithi029 bash[79294]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5 2024-02-10T07:02:41.677 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:41 smithi029 podman[79313]: 2024-02-10 07:02:41.406222169 +0000 UTC m=+0.679615192 container remove ea4052b4cafaafbfa0cc5018e0d297731b2f70e782a7df7ca641843e67be91ad (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-10T07:02:41.677 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:41 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:41.515+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:41.677 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:41 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:41.515+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:41.678 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:41 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:41.515+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:15.200213+0000 front 2024-02-10T07:02:15.200126+0000 (oldest deadline 2024-02-10T07:02:41.100058+0000) 2024-02-10T07:02:41.958 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:41 smithi029 podman[79405]: 2024-02-10 07:02:41.710572683 +0000 UTC m=+0.018291185 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:02:41.958 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:41 smithi029 podman[79405]: 2024-02-10 07:02:41.824796086 +0000 UTC m=+0.132514580 container create 43f399d1303b83511bc83d953c4c154562a3bf733e986361fc03eeaa72271cc0 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-10T07:02:41.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:41 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:41.676+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:41.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:41 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:41.676+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.944306+0000 front 2024-02-10T07:02:04.944329+0000 (oldest deadline 2024-02-10T07:02:33.144779+0000) 2024-02-10T07:02:41.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:41 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:41.676+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:14.245758+0000 front 2024-02-10T07:02:14.245691+0000 (oldest deadline 2024-02-10T07:02:38.945657+0000) 2024-02-10T07:02:42.214 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:42 smithi029 podman[79405]: 2024-02-10 07:02:42.095868773 +0000 UTC m=+0.403587268 container init 43f399d1303b83511bc83d953c4c154562a3bf733e986361fc03eeaa72271cc0 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-10T07:02:42.214 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:42 smithi029 podman[79405]: 2024-02-10 07:02:42.098956502 +0000 UTC m=+0.406674998 container start 43f399d1303b83511bc83d953c4c154562a3bf733e986361fc03eeaa72271cc0 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-10T07:02:42.214 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:42 smithi029 podman[79405]: 2024-02-10 07:02:42.214676115 +0000 UTC m=+0.522394622 container attach 43f399d1303b83511bc83d953c4c154562a3bf733e986361fc03eeaa72271cc0 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, ceph=True) 2024-02-10T07:02:42.477 INFO:journalctl@ceph.osd.5.smithi029.stdout:Feb 10 07:02:42 smithi029 podman[79405]: 2024-02-10 07:02:42.396713641 +0000 UTC m=+0.704432140 container died 43f399d1303b83511bc83d953c4c154562a3bf733e986361fc03eeaa72271cc0 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-10T07:02:42.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:42 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:42.477+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:42.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:42 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:42.477+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:42.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:42 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:42.477+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:15.200213+0000 front 2024-02-10T07:02:15.200126+0000 (oldest deadline 2024-02-10T07:02:41.100058+0000) 2024-02-10T07:02:42.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:42 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:42.670+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:42.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:42 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:42.670+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.944306+0000 front 2024-02-10T07:02:04.944329+0000 (oldest deadline 2024-02-10T07:02:33.144779+0000) 2024-02-10T07:02:42.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:42 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:42.670+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:14.245758+0000 front 2024-02-10T07:02:14.245691+0000 (oldest deadline 2024-02-10T07:02:38.945657+0000) 2024-02-10T07:02:43.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.5.service' 2024-02-10T07:02:43.096 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:02:43.096 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-10T07:02:43.096 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-10T07:02:43.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.6 2024-02-10T07:02:43.427 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:43 smithi029 systemd[1]: Stopping Ceph osd.6 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T07:02:43.427 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:43 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:43.426+0000 7f5cc8bfb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:02:43.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:43 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:43.515+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:43.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:43 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:43.515+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:43.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:43 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:43.515+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:15.200213+0000 front 2024-02-10T07:02:15.200126+0000 (oldest deadline 2024-02-10T07:02:41.100058+0000) 2024-02-10T07:02:43.708 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:43 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:43.426+0000 7f5cc8bfb700 -1 osd.6 142 *** Got signal Terminated *** 2024-02-10T07:02:43.708 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:43 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:43.426+0000 7f5cc8bfb700 -1 osd.6 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:02:43.708 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:43 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:43.650+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:43.708 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:43 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:43.650+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.944306+0000 front 2024-02-10T07:02:04.944329+0000 (oldest deadline 2024-02-10T07:02:33.144779+0000) 2024-02-10T07:02:43.709 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:43 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:43.650+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:14.245758+0000 front 2024-02-10T07:02:14.245691+0000 (oldest deadline 2024-02-10T07:02:38.945657+0000) 2024-02-10T07:02:44.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:44 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:44.505+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:44.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:44 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:44.505+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:44.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:44 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:44.505+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:15.200213+0000 front 2024-02-10T07:02:15.200126+0000 (oldest deadline 2024-02-10T07:02:41.100058+0000) 2024-02-10T07:02:44.959 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:44 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:44.609+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:44.959 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:44 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:44.609+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.944306+0000 front 2024-02-10T07:02:04.944329+0000 (oldest deadline 2024-02-10T07:02:33.144779+0000) 2024-02-10T07:02:44.959 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:44 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:44.609+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:14.245758+0000 front 2024-02-10T07:02:14.245691+0000 (oldest deadline 2024-02-10T07:02:38.945657+0000) 2024-02-10T07:02:45.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:45 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:45.504+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:45.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:45 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:45.504+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:45.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:45 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:45.504+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:15.200213+0000 front 2024-02-10T07:02:15.200126+0000 (oldest deadline 2024-02-10T07:02:41.100058+0000) 2024-02-10T07:02:45.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:45 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:45.611+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:45.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:45 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:45.611+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.944306+0000 front 2024-02-10T07:02:04.944329+0000 (oldest deadline 2024-02-10T07:02:33.144779+0000) 2024-02-10T07:02:45.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:45 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:45.611+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:14.245758+0000 front 2024-02-10T07:02:14.245691+0000 (oldest deadline 2024-02-10T07:02:38.945657+0000) 2024-02-10T07:02:45.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:45 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:45.611+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:02:21.846553+0000 front 2024-02-10T07:02:21.846567+0000 (oldest deadline 2024-02-10T07:02:45.346456+0000) 2024-02-10T07:02:46.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:46 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:46.587+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:46.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:46 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:46.587+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.944306+0000 front 2024-02-10T07:02:04.944329+0000 (oldest deadline 2024-02-10T07:02:33.144779+0000) 2024-02-10T07:02:46.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:46 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:46.587+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:14.245758+0000 front 2024-02-10T07:02:14.245691+0000 (oldest deadline 2024-02-10T07:02:38.945657+0000) 2024-02-10T07:02:46.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:46 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:46.587+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:02:21.846553+0000 front 2024-02-10T07:02:21.846567+0000 (oldest deadline 2024-02-10T07:02:45.346456+0000) 2024-02-10T07:02:46.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:46 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:46.467+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:46.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:46 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:46.467+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:46.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:46 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:46.467+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:15.200213+0000 front 2024-02-10T07:02:15.200126+0000 (oldest deadline 2024-02-10T07:02:41.100058+0000) 2024-02-10T07:02:46.959 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:46 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:46.467+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:02:21.100491+0000 front 2024-02-10T07:02:21.100512+0000 (oldest deadline 2024-02-10T07:02:46.300658+0000) 2024-02-10T07:02:47.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:47 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:47.482+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:47.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:47 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:47.482+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:47.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:47 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:47.482+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:15.200213+0000 front 2024-02-10T07:02:15.200126+0000 (oldest deadline 2024-02-10T07:02:41.100058+0000) 2024-02-10T07:02:47.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:47 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:47.482+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:02:21.100491+0000 front 2024-02-10T07:02:21.100512+0000 (oldest deadline 2024-02-10T07:02:46.300658+0000) 2024-02-10T07:02:47.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:47 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:47.594+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:59.242878+0000 front 2024-02-10T07:01:59.242750+0000 (oldest deadline 2024-02-10T07:02:21.542731+0000) 2024-02-10T07:02:47.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:47 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:47.594+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:04.944306+0000 front 2024-02-10T07:02:04.944329+0000 (oldest deadline 2024-02-10T07:02:33.144779+0000) 2024-02-10T07:02:47.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:47 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:47.594+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:14.245758+0000 front 2024-02-10T07:02:14.245691+0000 (oldest deadline 2024-02-10T07:02:38.945657+0000) 2024-02-10T07:02:47.958 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:47 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6[62649]: 2024-02-10T07:02:47.594+0000 7f5cc51ef700 -1 osd.6 142 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:02:21.846553+0000 front 2024-02-10T07:02:21.846567+0000 (oldest deadline 2024-02-10T07:02:45.346456+0000) 2024-02-10T07:02:48.835 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:48 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:48.521+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:48.835 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:48 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:48.521+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:48.835 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:48 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:48.521+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:15.200213+0000 front 2024-02-10T07:02:15.200126+0000 (oldest deadline 2024-02-10T07:02:41.100058+0000) 2024-02-10T07:02:48.835 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:48 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:48.521+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:02:21.100491+0000 front 2024-02-10T07:02:21.100512+0000 (oldest deadline 2024-02-10T07:02:46.300658+0000) 2024-02-10T07:02:49.139 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:48 smithi029 podman[79511]: 2024-02-10 07:02:48.834607288 +0000 UTC m=+5.516807456 container died d89cf7ba8295c30314e43a79b85e649b63eb0b5a66afe4a91b45bc69b350b6b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-10T07:02:49.458 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:49 smithi029 podman[79511]: 2024-02-10 07:02:49.254181172 +0000 UTC m=+5.936381354 container cleanup d89cf7ba8295c30314e43a79b85e649b63eb0b5a66afe4a91b45bc69b350b6b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2) 2024-02-10T07:02:49.458 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:49 smithi029 bash[79511]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6 2024-02-10T07:02:49.878 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:49 smithi029 podman[79528]: 2024-02-10 07:02:49.458079004 +0000 UTC m=+0.624073342 container remove d89cf7ba8295c30314e43a79b85e649b63eb0b5a66afe4a91b45bc69b350b6b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2) 2024-02-10T07:02:49.878 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:49 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:49.484+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:49.878 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:49 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:49.484+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:49.879 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:49 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:49.484+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:15.200213+0000 front 2024-02-10T07:02:15.200126+0000 (oldest deadline 2024-02-10T07:02:41.100058+0000) 2024-02-10T07:02:49.879 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:49 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:49.484+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:02:21.100491+0000 front 2024-02-10T07:02:21.100512+0000 (oldest deadline 2024-02-10T07:02:46.300658+0000) 2024-02-10T07:02:50.132 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:49 smithi029 podman[79616]: 2024-02-10 07:02:49.780208037 +0000 UTC m=+0.017529348 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:02:50.132 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:49 smithi029 podman[79616]: 2024-02-10 07:02:49.884580927 +0000 UTC m=+0.121902218 container create 38923721599f57882cad68e34892687d671498907e96092b9775ee8dee355424 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-deactivate, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-10T07:02:50.133 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:50 smithi029 podman[79616]: 2024-02-10 07:02:50.132620529 +0000 UTC m=+0.369941821 container init 38923721599f57882cad68e34892687d671498907e96092b9775ee8dee355424 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, RELEASE=main-290212a) 2024-02-10T07:02:50.431 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:50 smithi029 podman[79616]: 2024-02-10 07:02:50.135907265 +0000 UTC m=+0.373228565 container start 38923721599f57882cad68e34892687d671498907e96092b9775ee8dee355424 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-02-10T07:02:50.431 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:50 smithi029 podman[79616]: 2024-02-10 07:02:50.222916229 +0000 UTC m=+0.460237540 container attach 38923721599f57882cad68e34892687d671498907e96092b9775ee8dee355424 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:02:50.708 INFO:journalctl@ceph.osd.6.smithi029.stdout:Feb 10 07:02:50 smithi029 podman[79616]: 2024-02-10 07:02:50.431691039 +0000 UTC m=+0.669012338 container died 38923721599f57882cad68e34892687d671498907e96092b9775ee8dee355424 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240131, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.2) 2024-02-10T07:02:50.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:50 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:50.466+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:50.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:50 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:50.466+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:50.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:50 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:50.466+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:15.200213+0000 front 2024-02-10T07:02:15.200126+0000 (oldest deadline 2024-02-10T07:02:41.100058+0000) 2024-02-10T07:02:50.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:50 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:50.466+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:02:21.100491+0000 front 2024-02-10T07:02:21.100512+0000 (oldest deadline 2024-02-10T07:02:46.300658+0000) 2024-02-10T07:02:51.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.6.service' 2024-02-10T07:02:51.094 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:02:51.094 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-10T07:02:51.094 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-10T07:02:51.094 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.7 2024-02-10T07:02:51.444 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:51 smithi029 systemd[1]: Stopping Ceph osd.7 for a5f941ec-c7de-11ee-95b8-87774f69a715... 2024-02-10T07:02:51.444 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:51 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:51.441+0000 7f828acf3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:02:51.444 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:51 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:51.441+0000 7f828acf3700 -1 osd.7 142 *** Got signal Terminated *** 2024-02-10T07:02:51.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:51 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:51.441+0000 7f828acf3700 -1 osd.7 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:02:51.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:51 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:51.472+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:51.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:51 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:51.472+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:51.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:51 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:51.472+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:15.200213+0000 front 2024-02-10T07:02:15.200126+0000 (oldest deadline 2024-02-10T07:02:41.100058+0000) 2024-02-10T07:02:51.708 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:51 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:51.472+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:02:21.100491+0000 front 2024-02-10T07:02:21.100512+0000 (oldest deadline 2024-02-10T07:02:46.300658+0000) 2024-02-10T07:02:52.864 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:52 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:52.502+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:52.864 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:52 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:52.502+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:52.865 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:52 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:52.502+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:15.200213+0000 front 2024-02-10T07:02:15.200126+0000 (oldest deadline 2024-02-10T07:02:41.100058+0000) 2024-02-10T07:02:52.865 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:52 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:52.502+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:02:21.100491+0000 front 2024-02-10T07:02:21.100512+0000 (oldest deadline 2024-02-10T07:02:46.300658+0000) 2024-02-10T07:02:53.797 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:53 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:53.482+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:53.797 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:53 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:53.482+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:53.797 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:53 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:53.482+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:15.200213+0000 front 2024-02-10T07:02:15.200126+0000 (oldest deadline 2024-02-10T07:02:41.100058+0000) 2024-02-10T07:02:53.798 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:53 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:53.482+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:02:21.100491+0000 front 2024-02-10T07:02:21.100512+0000 (oldest deadline 2024-02-10T07:02:46.300658+0000) 2024-02-10T07:02:53.798 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:53 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:53.482+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.29:6804 osd.4 since back 2024-02-10T07:02:27.401326+0000 front 2024-02-10T07:02:27.401385+0000 (oldest deadline 2024-02-10T07:02:52.701288+0000) 2024-02-10T07:02:54.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:54 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:54.507+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:54.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:54 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:54.507+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:54.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:54 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:54.507+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:15.200213+0000 front 2024-02-10T07:02:15.200126+0000 (oldest deadline 2024-02-10T07:02:41.100058+0000) 2024-02-10T07:02:54.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:54 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:54.507+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:02:21.100491+0000 front 2024-02-10T07:02:21.100512+0000 (oldest deadline 2024-02-10T07:02:46.300658+0000) 2024-02-10T07:02:54.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:54 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:54.507+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.29:6804 osd.4 since back 2024-02-10T07:02:27.401326+0000 front 2024-02-10T07:02:27.401385+0000 (oldest deadline 2024-02-10T07:02:52.701288+0000) 2024-02-10T07:02:55.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:55 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:55.516+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:01:58.798778+0000 front 2024-02-10T07:01:58.798600+0000 (oldest deadline 2024-02-10T07:02:23.498560+0000) 2024-02-10T07:02:55.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:55 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:55.516+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:02:07.599482+0000 front 2024-02-10T07:02:07.599574+0000 (oldest deadline 2024-02-10T07:02:31.699290+0000) 2024-02-10T07:02:55.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:55 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:55.516+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:02:15.200213+0000 front 2024-02-10T07:02:15.200126+0000 (oldest deadline 2024-02-10T07:02:41.100058+0000) 2024-02-10T07:02:55.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:55 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:55.516+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:02:21.100491+0000 front 2024-02-10T07:02:21.100512+0000 (oldest deadline 2024-02-10T07:02:46.300658+0000) 2024-02-10T07:02:55.958 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:55 smithi029 ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7[64920]: 2024-02-10T07:02:55.516+0000 7f82832df700 -1 osd.7 142 heartbeat_check: no reply from 172.21.15.29:6804 osd.4 since back 2024-02-10T07:02:27.401326+0000 front 2024-02-10T07:02:27.401385+0000 (oldest deadline 2024-02-10T07:02:52.701288+0000) 2024-02-10T07:02:57.156 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:56 smithi029 podman[79722]: 2024-02-10 07:02:56.84678307 +0000 UTC m=+5.526751090 container died 156ce7027d7a10da781c50003137af43f916f588bee528a7b927289d5be4da15 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:02:57.458 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:57 smithi029 podman[79722]: 2024-02-10 07:02:57.313981141 +0000 UTC m=+5.993949162 container cleanup 156ce7027d7a10da781c50003137af43f916f588bee528a7b927289d5be4da15 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, ceph=True, RELEASE=main-290212a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:02:57.458 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:57 smithi029 bash[79722]: ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7 2024-02-10T07:02:57.938 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:57 smithi029 podman[79782]: 2024-02-10 07:02:57.583189755 +0000 UTC m=+0.740536035 container remove 156ce7027d7a10da781c50003137af43f916f588bee528a7b927289d5be4da15 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-10T07:02:58.198 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:57 smithi029 podman[79874]: 2024-02-10 07:02:57.8414243 +0000 UTC m=+0.017661585 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:02:58.198 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:57 smithi029 podman[79874]: 2024-02-10 07:02:57.959751978 +0000 UTC m=+0.135989302 container create b8375c6ff255616c37bf0bb2f8cc7f58cfdb1aae7de5e2681d23d833fd9126c2 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-deactivate, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131) 2024-02-10T07:02:58.199 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:58 smithi029 podman[79874]: 2024-02-10 07:02:58.198763708 +0000 UTC m=+0.375000993 container init b8375c6ff255616c37bf0bb2f8cc7f58cfdb1aae7de5e2681d23d833fd9126c2 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-10T07:02:58.458 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:58 smithi029 podman[79874]: 2024-02-10 07:02:58.202015056 +0000 UTC m=+0.378252347 container start b8375c6ff255616c37bf0bb2f8cc7f58cfdb1aae7de5e2681d23d833fd9126c2 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-deactivate, RELEASE=main-290212a, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-10T07:02:58.458 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:58 smithi029 podman[79874]: 2024-02-10 07:02:58.307613209 +0000 UTC m=+0.483850504 container attach b8375c6ff255616c37bf0bb2f8cc7f58cfdb1aae7de5e2681d23d833fd9126c2 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, maintainer=Guillaume Abrioux ) 2024-02-10T07:02:58.878 INFO:journalctl@ceph.osd.7.smithi029.stdout:Feb 10 07:02:58 smithi029 podman[79874]: 2024-02-10 07:02:58.502598124 +0000 UTC m=+0.678835406 container died b8375c6ff255616c37bf0bb2f8cc7f58cfdb1aae7de5e2681d23d833fd9126c2 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-a5f941ec-c7de-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-10T07:02:59.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a5f941ec-c7de-11ee-95b8-87774f69a715@osd.7.service' 2024-02-10T07:02:59.262 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:02:59.262 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-10T07:02:59.263 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T07:05:56.199 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T07:06:17.941 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:06:17.978 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:06:18.006 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-10T07:06:18.008 DEBUG:teuthology.misc:Transferring archived files from smithi008:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555158/remote/smithi008/crash 2024-02-10T07:06:18.009 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/crash -- . 2024-02-10T07:06:18.057 INFO:teuthology.orchestra.run.smithi008.stderr:tar: /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T07:06:18.057 INFO:teuthology.orchestra.run.smithi008.stderr:tar: Error is not recoverable: exiting now 2024-02-10T07:06:18.059 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555158/remote/smithi029/crash 2024-02-10T07:06:18.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/crash -- . 2024-02-10T07:06:18.099 INFO:teuthology.orchestra.run.smithi029.stderr:tar: /var/lib/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T07:06:18.100 INFO:teuthology.orchestra.run.smithi029.stderr:tar: Error is not recoverable: exiting now 2024-02-10T07:06:18.100 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-10T07:06:18.101 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T07:06:18.133 INFO:teuthology.orchestra.run.smithi008.stdout:2024-02-10T06:41:50.788539+0000 mon.smithi008 (mon.0) 494 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T07:06:18.133 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-10T07:06:18.133 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[SEC\]' /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T07:06:18.205 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[ERR\]' /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T07:06:18.276 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[WRN\]' /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T07:06:18.346 INFO:teuthology.orchestra.run.smithi008.stdout:2024-02-10T06:41:50.788539+0000 mon.smithi008 (mon.0) 494 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T07:06:18.347 INFO:tasks.cephadm:Compressing logs... 2024-02-10T07:06:18.347 DEBUG:teuthology.orchestra.run.smithi008:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T07:06:18.389 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T07:06:18.416 INFO:teuthology.orchestra.run.smithi008.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-10T07:06:18.416 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T07:06:18.416 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mds.foofs.smithi008.vfwcte.log 2024-02-10T07:06:18.416 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T07:06:18.416 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mds.foofs.smithi008.vfwcte.log: gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mon.smithi008.log 2024-02-10T07:06:18.416 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.1.log 2024-02-10T07:06:18.416 INFO:teuthology.orchestra.run.smithi008.stderr: 89.4% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mds.foofs.smithi008.vfwcte.log.gz 2024-02-10T07:06:18.416 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.3.log 2024-02-10T07:06:18.417 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mon.smithi008.log: /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.0.log 2024-02-10T07:06:18.417 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T07:06:18.420 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.log 2024-02-10T07:06:18.420 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.2.log 2024-02-10T07:06:18.420 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.log: 85.8% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T07:06:18.420 INFO:teuthology.orchestra.run.smithi029.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T07:06:18.421 INFO:teuthology.orchestra.run.smithi029.stderr:: No such file or directory 2024-02-10T07:06:18.421 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.6.log 2024-02-10T07:06:18.421 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mon.smithi029.log 2024-02-10T07:06:18.421 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T07:06:18.421 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mon.smithi029.log: gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.4.log 2024-02-10T07:06:18.421 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.7.log 2024-02-10T07:06:18.421 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.5.log 2024-02-10T07:06:18.421 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mgr.smithi029.fuvrtf.log 2024-02-10T07:06:18.421 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mds.foofs.smithi029.bwmcyt.log 2024-02-10T07:06:18.423 INFO:teuthology.orchestra.run.smithi008.stderr: 91.5% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T07:06:18.423 INFO:teuthology.orchestra.run.smithi008.stderr: 87.7% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T07:06:18.423 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mgr.smithi008.pxifbt.log 2024-02-10T07:06:18.423 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T07:06:18.425 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mgr.smithi029.fuvrtf.log: 91.6% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T07:06:18.425 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T07:06:18.425 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mds.foofs.smithi029.bwmcyt.log: 87.6% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mds.foofs.smithi029.bwmcyt.log.gz 2024-02-10T07:06:18.425 INFO:teuthology.orchestra.run.smithi029.stderr: 90.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T07:06:18.425 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.log 2024-02-10T07:06:18.425 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T07:06:18.425 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.log: 85.8% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T07:06:18.428 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-volume.log: 87.7% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T07:06:18.431 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mgr.smithi008.pxifbt.log: /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-volume.log: 89.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T07:06:18.438 INFO:teuthology.orchestra.run.smithi008.stderr: 91.4% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T07:06:18.441 INFO:teuthology.orchestra.run.smithi029.stderr: 91.5% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T07:06:18.483 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mgr.smithi029.fuvrtf.log.gz 2024-02-10T07:06:18.711 INFO:teuthology.orchestra.run.smithi008.stderr: 88.3% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mgr.smithi008.pxifbt.log.gz 2024-02-10T07:06:18.774 INFO:teuthology.orchestra.run.smithi029.stderr: 92.6% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mon.smithi029.log.gz 2024-02-10T07:06:19.834 INFO:teuthology.orchestra.run.smithi008.stderr: 91.9% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-mon.smithi008.log.gz 2024-02-10T07:06:31.346 INFO:teuthology.orchestra.run.smithi008.stderr: 94.5% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.2.log.gz 2024-02-10T07:06:34.311 INFO:teuthology.orchestra.run.smithi029.stderr: 93.9% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.4.log.gz 2024-02-10T07:06:36.134 INFO:teuthology.orchestra.run.smithi029.stderr: 94.2% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.6.log.gz 2024-02-10T07:06:36.317 INFO:teuthology.orchestra.run.smithi008.stderr: 94.1% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.0.log.gz 2024-02-10T07:06:37.433 INFO:teuthology.orchestra.run.smithi008.stderr: 94.2% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.1.log.gz 2024-02-10T07:06:39.478 INFO:teuthology.orchestra.run.smithi029.stderr: 94.1% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.5.log.gz 2024-02-10T07:06:39.980 INFO:teuthology.orchestra.run.smithi008.stderr: 94.1% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.3.log.gz 2024-02-10T07:06:39.983 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-02-10T07:06:39.983 INFO:teuthology.orchestra.run.smithi008.stderr:real 0m21.581s 2024-02-10T07:06:39.983 INFO:teuthology.orchestra.run.smithi008.stderr:user 1m10.481s 2024-02-10T07:06:39.983 INFO:teuthology.orchestra.run.smithi008.stderr:sys 0m2.571s 2024-02-10T07:06:45.528 INFO:teuthology.orchestra.run.smithi029.stderr: 94.0% -- replaced with /var/log/ceph/a5f941ec-c7de-11ee-95b8-87774f69a715/ceph-osd.7.log.gz 2024-02-10T07:06:45.530 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-02-10T07:06:45.530 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m27.127s 2024-02-10T07:06:45.530 INFO:teuthology.orchestra.run.smithi029.stderr:user 1m19.105s 2024-02-10T07:06:45.530 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m2.848s 2024-02-10T07:06:45.531 INFO:tasks.cephadm:Archiving logs... 2024-02-10T07:06:45.531 DEBUG:teuthology.misc:Transferring archived files from smithi008:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555158/remote/smithi008/log 2024-02-10T07:06:45.532 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T07:06:55.330 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555158/remote/smithi029/log 2024-02-10T07:06:55.332 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T07:07:06.326 INFO:tasks.cephadm:Removing cluster... 2024-02-10T07:07:06.327 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 --force 2024-02-10T07:07:06.711 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a5f941ec-c7de-11ee-95b8-87774f69a715 --force 2024-02-10T07:07:07.107 INFO:tasks.cephadm:Removing cephadm ... 2024-02-10T07:07:07.107 DEBUG:teuthology.orchestra.run.smithi008:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T07:07:07.122 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T07:07:07.136 INFO:tasks.cephadm:Teardown complete 2024-02-10T07:07:07.136 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-10T07:07:07.147 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-10T07:07:07.147 DEBUG:teuthology.orchestra.run.smithi008:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T07:07:07.166 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T07:07:07.177 INFO:teuthology.orchestra.run.smithi008.stderr:bash: line 1: ntpq: command not found 2024-02-10T07:07:07.181 INFO:teuthology.orchestra.run.smithi008.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T07:07:07.181 INFO:teuthology.orchestra.run.smithi008.stdout:=============================================================================== 2024-02-10T07:07:07.181 INFO:teuthology.orchestra.run.smithi008.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +918us[ +918us] +/- 99ms 2024-02-10T07:07:07.181 INFO:teuthology.orchestra.run.smithi008.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 -803us[ -778us] +/- 46ms 2024-02-10T07:07:07.182 INFO:teuthology.orchestra.run.smithi008.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +562us[ +562us] +/- 74ms 2024-02-10T07:07:07.182 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:07:07.189 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-02-10T07:07:07.194 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T07:07:07.194 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-02-10T07:07:07.194 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 55 +906us[ +919us] +/- 98ms 2024-02-10T07:07:07.194 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -800us[ -787us] +/- 46ms 2024-02-10T07:07:07.194 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +555us[ +568us] +/- 74ms 2024-02-10T07:07:07.194 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:07:07.194 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-10T07:07:07.204 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-10T07:07:07.205 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-10T07:07:07.228 DEBUG:teuthology.orchestra.run.smithi008:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T07:07:07.313 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T07:07:07.391 DEBUG:teuthology.orchestra.run.smithi008:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T07:07:07.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:07:07.422 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T07:07:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:07:07.452 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-10T07:07:07.462 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-10T07:07:07.495 INFO:teuthology.task.internal:Duration was 2134.385322 seconds 2024-02-10T07:07:07.495 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-10T07:07:07.528 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-10T07:07:07.528 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T07:07:07.530 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T07:07:07.571 INFO:teuthology.orchestra.run.smithi008.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T07:07:07.572 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T07:07:08.077 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-10T07:07:08.077 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T07:07:08.077 DEBUG:teuthology.orchestra.run.smithi008:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T07:07:08.137 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2024-02-10T07:07:08.138 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T07:07:08.167 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-10T07:07:08.167 DEBUG:teuthology.orchestra.run.smithi008:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T07:07:08.180 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T07:07:08.237 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-10T07:07:08.237 DEBUG:teuthology.orchestra.run.smithi008:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T07:07:08.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T07:07:22.306 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-10T07:07:22.317 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-10T07:07:22.318 DEBUG:teuthology.orchestra.run.smithi008:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T07:07:22.345 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T07:07:22.377 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-10T07:07:22.387 DEBUG:teuthology.orchestra.run.smithi008:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T07:07:22.390 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T07:07:22.412 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern = core 2024-02-10T07:07:22.446 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-02-10T07:07:22.473 DEBUG:teuthology.orchestra.run.smithi008:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T07:07:22.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:07:22.510 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T07:07:22.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:07:22.528 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-10T07:07:22.539 INFO:teuthology.task.internal:Transferring archived files... 2024-02-10T07:07:22.540 DEBUG:teuthology.misc:Transferring archived files from smithi008:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555158/remote/smithi008 2024-02-10T07:07:22.540 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T07:07:22.616 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555158/remote/smithi029 2024-02-10T07:07:22.616 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T07:07:22.658 INFO:teuthology.task.internal:Removing archive directory... 2024-02-10T07:07:22.658 DEBUG:teuthology.orchestra.run.smithi008:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T07:07:22.661 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T07:07:22.708 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-10T07:07:22.719 INFO:teuthology.task.internal:Not uploading archives. 2024-02-10T07:07:22.719 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-10T07:07:22.813 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-10T07:07:22.814 DEBUG:teuthology.orchestra.run.smithi008:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T07:07:22.816 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T07:07:22.828 INFO:teuthology.orchestra.run.smithi008.stdout: 265787 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 07:07 /home/ubuntu/cephtest 2024-02-10T07:07:22.831 INFO:teuthology.orchestra.run.smithi029.stdout: 265293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 07:07 /home/ubuntu/cephtest 2024-02-10T07:07:22.832 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-10T07:07:22.843 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-10T07:07:22.920 INFO:teuthology.nuke:Checking targets against current locks 2024-02-10T07:07:22.951 DEBUG:teuthology.nuke:shortname: smithi008 2024-02-10T07:07:22.951 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T07:07:22.982 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi008.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555158', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 06:22:27.726461', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF2aUKsBzj/VJ8KwhUWNGSToLomQpKaI08g8aqfn3rmIfB0jYyt1/LwmV9XVr8zpa3p+ZtDByLoVxoE517vK6Y0='} 2024-02-10T07:07:22.983 DEBUG:teuthology.nuke:shortname: smithi029 2024-02-10T07:07:22.984 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T07:07:23.011 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555158', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 06:22:27.728218', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNrtBko9bMq22M2Y9UwpUu0ycburNrK0a2irK58FScViUNaLzP/mmR34uUX9Y6Hc/8xdrXUcoIreMYz8geIsGjA='} 2024-02-10T07:07:23.042 INFO:teuthology.orchestra.console.smithi008:Power off 2024-02-10T07:07:23.042 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T07:07:23.067 DEBUG:teuthology.orchestra.console.smithi008:power off output: Chassis Power Control: Down/Off 2024-02-10T07:07:23.067 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:07:23.084 INFO:teuthology.orchestra.console.smithi029:Power off 2024-02-10T07:07:23.084 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T07:07:23.094 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is on 2024-02-10T07:07:23.111 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2024-02-10T07:07:23.111 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:07:23.138 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-02-10T07:07:27.096 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:07:27.139 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:07:27.206 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is on 2024-02-10T07:07:27.249 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-02-10T07:07:31.208 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:07:31.250 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T07:07:31.319 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is off 2024-02-10T07:07:31.360 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2024-02-10T07:07:31.420 INFO:teuthology.orchestra.console.smithi008:Power off completed 2024-02-10T07:07:31.461 INFO:teuthology.orchestra.console.smithi029:Power off completed 2024-02-10T07:07:31.565 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 2134.385321855545 failure_reason: '"2024-02-10T06:41:50.788539+0000 mon.smithi008 (mon.0) 494 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log' owner: scheduled_yuriw@teuthology success: false 2024-02-10T07:07:31.566 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T07:07:31.646 INFO:teuthology.run:FAIL